E. Capacity and continuity¶
How the system keeps working under high load and partial failure.
Take the Capacity and continuity quiz when you have read the examples.
P44. Margin¶
The rule: design capacity above the expected load (margin), state the design load as a probability per time period, keep queued resources well below 100 % utilization, and set the margin by design maturity: larger early, smaller as uncertainty falls
Illustrative, not evidence.
Scenario. A ticketing service averages 400 requests per second. The busiest day in 100 reaches 900 requests per second. The team sizes its worker pool.
Applying the rule. The team states the design load as 900 requests per second, the 1-in-100-day peak, and adds 30 % headroom, so 1,170. It sizes the pool to stay below 70 % average utilization. A load test at 1,170 requests per second shows queue delay within the 200 ms target.
Violating the rule. The team sizes the pool exactly to the average of 400 requests per second. On the first busy day the queue grows, delay climbs sharply, and requests time out.
Management variant.
Scenario. A team lead plans the next sprint for a team that also handles unplanned support work.
Applying the rule. The lead plans 70 % of the team's capacity and keeps 30 % free for unplanned work. When two urgent bugs arrive, the sprint goal still holds.
Violating the rule. The lead plans every person at 100 %. Each urgent bug pushes planned work out, and the sprint misses its goal.
Real case: Hyatt Regency walkway collapse (1981).
On 17 July 1981, two suspended walkways in the atrium of the Hyatt Regency Hotel in Kansas City collapsed. The National Bureau of Standards found that the box beam-hanger rod connections did not meet the Kansas City Building Code, even in the original design. A change of the hanger rods during construction about doubled the load on the fourth floor connections. The walkways fell under loads well below the design loads of the code.
What it shows for P44. At the collapse, the load on a connection was only 31 percent of the capacity the code expected. The walkways had almost no capacity beyond their own weight, so there was no margin above the expected load.
On July 17, 1981, two suspended walkways within the atrium area of the Hyatt Regency Hotel in Kansas City, Mo., collapsed, leaving 113 people dead and 186 injured.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
Two factors contributed to the collapse: inadequacy of the original design for the box beam-hanger rod connection, which was identical for all three walkways, and a change in hanger rod arrangement during construction that essentially doubled the load on the box beam-hanger rod connections at the fourth floor walkway.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
the maximum load on a fourth floor box beam-hanger rod connection at the time of collapse was only 31 percent of the ultimate capacity expected of a connection designed under the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
they had only minimal capacity to resist their own weight and had virtually no capacity to resist additional loads imposed by people.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
Collapse of the walkways occurred under the action of loads that were substantially less than the design loads specified by the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
As constructed, the box beam-hanger rod connections, the fourth floor to ceiling hanger rods, and the third floor walkway hanger rods did not satisfy the design provisions of the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
The box beam-hanger rod connection would not have satisfied the Kansas City Building Code under the original hanger rod detail (continuous rod)
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
P45. Redundancy (N-1)¶
The rule: add redundancy, so that the function survives the loss of any single element (N-1), and make sure each copy and the switch-over mechanism are reliable enough for the redundancy to help
Illustrative, not evidence.
Scenario. A payments API is a critical service. One failed node must not stop it.
Applying the rule. The API runs on three nodes in three availability zones. The switch-over is a load balancer with health checks, and it is tested too. In a test the team removes one node at a time, and requests keep succeeding each time.
Violating the rule. The queue service behind the API runs as a single replica. When its node is replaced during maintenance, payments stop until it restarts.
Management variant.
Scenario. A team runs an on-call rota. Some skills are needed for every incident.
Applying the rule. Each on-call skill is held by at least two named people, and each has a trained successor. If one person leaves or falls ill, the function continues.
Violating the rule. Only one person knows how to restore the billing batch job, and nothing is documented. When that person is on leave, a failure stays unfixed for days.
Real case: The 2003 Northeast blackout (2003).
On 14 August 2003, a blackout hit an area with about 50 million people in the United States and Canada. At FirstEnergy (FE), the control room alarm software failed. The alarm application then failed over, still stalled, to the back-up server, which failed too. Operators lost their alarms while lines tripped. FE had no automatic load-shedding schemes and did not begin manual load shedding. The task force found that shedding 1,500 MW in the Cleveland-Akron area before a key line outage could have averted the blackout.
What it shows for P45. The alarm system had a back-up server, but the stalled application moved to it intact and the back-up failed too. A back-up only helps if the switch-over does not carry the failure with it, so the switch-over has to be tested under a fault, not only a clean shutdown.
The outage affected an area with an estimated 50 million people and 61,800 megawatts (MW) of electric load
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
Shortly after 14:14 EDT, the alarm and logging system in FE’s control room failed and was not restored until after the blackout.
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
However, because the alarm application moved intact onto the backup while still stalled and ineffective, the backup server failed 13 minutes later, at 14:54 EDT.
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
FirstEnergy had no automatic load-shedding schemes in place, and did not attempt to begin manual load-shedding.
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
if manual or automatic load-shedding of 1,500 MW had occurred within the Cleveland-Akron area before that outage, the blackout could have been averted.
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
A cascade is a dynamic phenomenon that cannot be stopped by human intervention once started.
Source: U.S.-Canada Power System Outage Task Force: Final Report on the August 14, 2003 Blackout (2004)
P46. Diversity¶
The rule: make the copies different (diversity)
Illustrative, not evidence.
Scenario. A team runs three replicas of a login service. They must survive causes that hit all replicas at once.
Applying the rule. The team lists the shared dependencies of the replicas: the cloud region, the base image, the TLS certificate and the DNS provider. Each one is either made different or justified in writing. The certificate is issued separately for each replica with staggered expiry dates, and two DNS providers serve the zone.
Violating the rule. All three replicas use the same TLS certificate. It expires on a Sunday, and all three fail together although there are three copies.
Management variant.
Scenario. An operations manager plans a status page and a review of a risky decision.
Applying the rule. The status page runs in two regions from two different providers. The risk review team includes people from another department who hold different views.
Violating the rule. The status page has two copies, but both run at the same provider in the same region. A review team of colleagues who think alike signs off the plan.
Real case: Ariane 5 Flight 501 failure (1996).
On 4 June 1996, about 40 seconds after the start of the flight sequence, the first Ariane 5 veered off its flight path, broke up and exploded. Its guidance used two inertial reference systems (SRIs) in parallel, with identical hardware and software, practically the same as on Ariane 4. Ariane 5's early trajectory gives much higher horizontal velocity than Ariane 4's, and a data conversion that was not protected failed on the larger value. The error came from an alignment function that serves no purpose after lift-off. The back-up SRI failed first, and the active SRI failed 0.05 seconds later for the same reason. The Ariane 5 trajectory data were not in the SRI specification, so the function was never tested under simulated Ariane 5 flight conditions.
What it shows for P46. The two SRIs were redundant copies, but they ran the same software. One design error hit both copies within 0.05 seconds, so the redundancy gave no protection. Copies that share a cause need to be made different, or the shared dependency needs to be justified.
On 4 June 1996 the maiden flight of the Ariane 5 launcher ended in a failure. Only about 40 seconds after initiation of the flight sequence, at an altitude of about 3700 m, the launcher veered off its flight path, broke up and exploded.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
The design of the Ariane 5 SRI is practically the same as that of an SRI which is presently used on Ariane 4, particularly as regards the software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
There are two SRIs operating in parallel, with identical hardware and software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The data conversion instructions (in Ada code) were not protected from causing an Operand Error, although other conversions of comparable variables in the same place in the code were protected.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
As soon as the launcher lifts off, this function serves no purpose.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The value of BH was much higher than expected because the early part of the trajectory of Ariane 5 differs from that of Ariane 4 and results in considerably higher horizontal velocity values.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
0.05 seconds later the active inertial reference system, identical to the back-up system in hardware and software, failed for the same reason.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
it was jointly agreed not to include the Ariane 5 trajectory data in the SRI requirements and specification.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
Consequently the realignment function was not tested under simulated Ariane 5 flight conditions, and the design error was not discovered.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
the equipment and system tests were not sufficiently representative.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
P47. Graceful degradation¶
The rule: define in advance which lower-priority functions switch off first, so that critical functions keep running (graceful degradation; opposite: cascading failure)
Illustrative, not evidence.
Scenario. An online shop expects demand to exceed capacity during a sale.
Applying the rule. Before the sale, the team defines the order in which functions switch off. Product recommendations go first, then search suggestions, and checkout runs last. An overload test shows low-priority functions stop first while checkout keeps working.
Violating the rule. Nothing is prioritized. Under sale traffic, every function slows down together, queues fill everywhere, and the whole shop fails, checkout included.
Management variant.
Scenario. A support organization can be hit by more requests than it can handle after an outage.
Applying the rule. The team decides in advance that feature requests stop first and that incident response and security patches continue. It also states the minimum capacity at which stopped activities resume.
Violating the rule. Every request type is handled in arrival order. During the outage, security patches wait behind feature requests.
Real case: The 2003 Northeast blackout (2003). Read the full case above.
What it shows for P47. FirstEnergy had no automatic load-shedding schemes and did not start manual load shedding. The task force found that shedding 1,500 MW in the Cleveland-Akron area before a key line outage could have averted the blackout. Load shedding planned in advance switches off some demand so the rest keeps running; once a cascade starts, people cannot stop it.
P48. Damping¶
The rule: add a deadband, delay or cooldown, so that the loop does not switch back and forth (oscillation; opposite: stable control)
Illustrative, not evidence.
Scenario. An autoscaler adds and removes workers based on CPU load measurements.
Applying the rule. The autoscaler has a deadband (scale up above 75 %, down below 45 %) and a five-minute cooldown. It is documented. A test with noisy load shows no repeated switching.
Violating the rule. The autoscaler scales up when CPU passes 60 % and down when it drops below 60 %. It adds and removes workers every minute and wastes capacity.
Management variant.
Scenario. A product organization reacts to customer feedback and market news.
Applying the rule. Priorities change on a fixed cadence, at sprint boundaries. A new signal waits for the next planning session unless it is a real emergency.
Violating the rule. Priorities change every day with each new signal. Teams start and drop work repeatedly and finish little.
P49. Induced demand¶
The rule: expect demand to grow into the new capacity, and also manage demand (quotas, priorities, pricing)
Illustrative, not evidence.
Scenario. A platform team's CI cluster is congested, and builds wait in a queue.
Applying the rule. The team doubles the runners. Before doing so it forecasts that demand will grow into the new capacity. It adds a quota per team and a lower priority for nightly builds. Both demand controls are in place with the capacity increase.
Violating the rule. The team doubles the cluster and changes nothing else. Faster CI leads to more CI runs, and the cluster is full again a month later.
Management variant.
Scenario. A department asks for more budget for a project that is behind.
Applying the rule. The extra budget comes with an explicit scope and a stated end date. The work is not allowed to grow to use up the money.
Violating the rule. The budget is raised with the words more of the same. The work expands to fill it, and the project is still not finished.
P50. Bottleneck first¶
The rule: find the constraint (bottleneck) and improve it first
Illustrative, not evidence.
Scenario. A team wants to ship changes faster. A change takes three days from first commit to release.
Applying the rule. Before choosing an improvement, the team measures each step. Lint takes 2 minutes, integration tests take 40 minutes, and changes wait about two days in the review queue. It names the review queue as the constraint and improves it first.
Violating the rule. The team spends a sprint making lint twice as fast. Work still piles up in front of the review queue, and the release time does not change.
Management variant.
Scenario. A company wants products to launch sooner. Every launch needs a security review.
Applying the rule. The security review queue is measured and named as the constraint. Review capacity is added there first.
Violating the rule. The company adds more developers, but the single security reviewer is the limit. Launches do not get faster.
P51. Shared demand signal¶
The rule: share the real end demand with every stage, instead of letting each stage react only to the stage below it
Illustrative, not evidence.
Scenario. A request passes through a front end, an API and a worker service. Each has its own autoscaler.
Applying the rule. The front end publishes its real traffic. The API and the worker service scale on that end-demand signal as well as on their own queue length. Every stage in the chain can see the end demand.
Violating the rule. Each autoscaler reacts only to its own queue. A small traffic bump grows into a large swing at the API and a larger one at the worker service.
Management variant.
Scenario. A platform team supports several product teams.
Applying the rule. The platform team sees the product teams' roadmaps and expected launches. It plans capacity for the real end demand.
Violating the rule. The platform team sees only the tickets that arrive. It reacts late to each launch, and its workload swings.
P52. Limit work in progress¶
The rule: set a limit on the number of items in progress, and finish items before starting new ones
Illustrative, not evidence.
Scenario. A team handles a flow of feature requests. It sets up a board.
Applying the rule. The board has a visible limit of four epics in progress. A new epic starts only when one finishes. The team tracks work in progress and lead time, and lead time falls.
Violating the rule. Everything is in progress and nothing is finished. There are 23 open epics for a team of six, and each waits weeks for attention.
P53. Late work and added people¶
The rule: do not expect added people to recover the schedule soon; cut or move scope first, and add people early, only where the work can be split
Illustrative, not evidence.
Scenario. A cluster migration is two weeks late, and a deadline is two weeks away.
Applying the rule. The lead cuts scope first, by moving two low-value services to the next migration. A record states the expected training and coordination cost of new people. New engineers join the next migration from its start, where the work can be split.
Violating the rule. The lead adds five engineers to the migration two weeks before the deadline. The existing team spends its time training them and the migration slips further.