C. Design¶
How to shape a design before it is built, so that it fails less.
Take the Design quiz when you have read the examples.
P22. Failure and interaction analysis¶
The rule: before you build, list how each part can fail, how parts that have not failed can interact unsafely, and how the system can cause harm while working as designed (insufficient design or foreseeable misuse)
Illustrative, not evidence.
Scenario. A team designs a job scheduler that depends on a registry, a queue and two controllers.
Applying the rule. Before building, the team lists how each part can fail and how the two controllers can fight over the same job when neither has failed. It also lists how a correct scheduler could still harm users through misuse. A reviewer signs off on the list for each component.
Violating the rule. The team builds the scheduler and ships it. The first failure analysis happens after an outage in which the registry went down and two controllers restarted the same jobs.
Management variant.
Scenario. A company plans to migrate three clusters in one quarter.
Applying the rule. Before the start, the teams assume the migration failed and list the reasons. They also list how the parallel migrations can interfere, and they find that two teams plan the same freeze window.
Violating the rule. Each team plans its own migration and reports only the benefits. Nobody asks how the migrations interfere, and two of them collide on the same freeze window.
Real case: Therac-25 radiation overdoses (1985).
Between June 1985 and January 1987, six known accidents involved massive overdoses by the Therac-25 radiation therapy machine. The machine relied more on software for safety than earlier models, and its maker chose not to duplicate all the old hardware interlocks. Its safety analysis, a fault tree, apparently excluded the software. Operators saw messages such as "Malfunction 54" that the machine's documentation did not explain.
What it shows for P22. The safety analysis left out the part that had taken over the safety functions. The software replaced hardware interlocks, but the fault tree apparently excluded it. A failure analysis must cover every part that can cause harm, software included.
Between June 1985 and January 1987, six known accidents involved massive overdoses by the Therac-25
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part I (1993)
the Therac-25 software has more responsibility for maintaining safety than the software in the previous machines.
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part I (1993)
decided not to duplicate all the existing hardware safety mechanisms and interlocks.
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part I (1993)
In March 1983, AECL performed a safety analysis on the Therac-25. This analysis was in the form of a fault tree and apparently excluded the software.
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part I (1993)
The ETCC did not have any other information available in its instruction manual or other Therac-25 documentation to explain the meaning of Malfunction 54.
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part II (1993)
The operator was accustomed to the quirks of the machine, which would frequently stop or delay treatment. In the past, the only consequences had been inconvenience.
Source: Leveson & Turner: An Investigation of the Therac-25 Accidents, Part II (1993)
P23. Simplicity and modularity¶
The rule: choose the simplest design that meets the requirements, and split it into modules
Illustrative, not evidence.
Scenario. A platform team packages ten applications for deployment.
Applying the rule. Each application has its own Helm chart with one stated purpose. A review finds no unused values, templates or flags.
Violating the rule. The team adds options for imagined future needs to one shared chart for all ten applications. Most options are unused, and a change for one application breaks another.
Management variant.
Scenario. A product team tracks its work.
Applying the rule. The team keeps one backlog with one ranked list and drops meetings that do not serve the objective.
Violating the rule. The team keeps three parallel priority lists and adds a new approval layer for imagined future needs. Nobody can say which work comes first.
Real case: Knight Capital trading loss (2012).
Knight Capital deployed new trading code to its order router, SMARS. The code reused a flag once used by an old function, Power Peg. Knight had stopped using Power Peg years earlier, but its code was still present and callable. A technician did not copy the new code to one of the eight servers, and no second technician reviewed the deployment. On 1 August 2012, in about 45 minutes, the old code on that server sent millions of orders into the market, while staff tried to find the source of the problem. Before the market opened, an internal system had sent 97 e-mails that pointed at the fault, but they were not designed as alerts. Knight lost 460 million dollars.
What it shows for P23. Unused code is not harmless while it can still run. Power Peg had no purpose, but it stayed in production and its flag was reused. When the flag was set, the dead code woke up. Remove parts that have no purpose.
On August 1, 2012, Knight Capital Americas LLC (“Knight”) experienced a significant error in the operation of its automated routing system for equity orders, known as SMARS.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
This unused code previously had been used for functionality called “Power Peg,” which Knight had discontinued using many years earlier.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Despite the lack of use, the Power Peg functionality remained present and callable at the time of the RLP deployment.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
The new RLP code also repurposed a flag that was formerly used to activate the Power Peg code.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Ultimately, Knight realized a $460 million loss on these positions.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
SMARS routed millions of orders into the market over a 45-minute period
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system sent 97 of these e-mail messages to a group of Knight personnel before the 9:30 a.m. market open.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system continued to send millions of child orders while its personnel attempted to identify the source of the problem.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight needed clear guidance for its technology personnel as to when to disconnect a malfunctioning system from the market.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
P24. Lifecycle and decommissioning¶
The rule: plan operation, maintenance and decommissioning at design time
Illustrative, not evidence.
Scenario. A team designs a new Kubernetes cluster for a data service.
Applying the rule. The design document has a maintenance section and a decommissioning section. The decommissioning section says how to delete the cluster and its data before the cluster is created.
Violating the rule. The design document ends at go-live. Years later, orphaned volumes and DNS records remain, and nobody can safely remove them.
Management variant.
Scenario. A company signs a contract with a hosting vendor.
Applying the rule. The contract has an exit clause and a data-return clause. The personal data collected under it has a deletion date.
Violating the rule. The contract says nothing about the end. When the company leaves, the vendor keeps its data and no one knows how to get it back.
Real case: Goiânia radiotherapy source (1987).
A private radiotherapy institute in Goiânia, Brazil, moved to new premises and left a caesium-137 teletherapy unit behind. It did not notify the licensing authority. People removed the strongly radioactive source from the abandoned machine, and it ruptured. Four people died and 28 suffered radiation burns.
What it shows for P24. The harm came after the clinic moved out and left the machine behind in the abandoned building. The source was still strongly radioactive. Planning decommissioning at design time gives a dangerous part a set way to be returned or disposed of when its use ends, instead of being left behind.
moved to new premises, taking with it a cobalt-60 teletherapy unit
without notifying the licensing authority as required under the terms of the institute's licence.
a shielded, strongly radioactive caesium-137 source (50.9 TBq, or 1375 Ci, at the time) was removed from its protective housing in a teletherapy machine in an abandoned clinic in Goiania, Brazil, and subsequently ruptured.
Four of the casualties ultimately died and 28 people suffered radiation burns.
P25. Human factors¶
The rule: design the interface for the real user in the real environment, including stress, and for the full range of users, not an average user
Illustrative, not evidence.
Scenario. An on-call engineer must fail over a database during a night incident.
Applying the rule. The team tests the failover runbook with real on-call engineers at night and under time pressure. The CLI output is unambiguous, and the team fixes the steps where testers slip.
Violating the rule. The failover command prints similar messages for success and failure. An engineer misreads one at 3 a.m., and the post-incident review blames the engineer.
Management variant.
Scenario. A team of eight starts a new project.
Applying the rule. Each member can name their role and the team's current goal. The workload is set at a level that people can carry.
Violating the rule. Roles are unclear and goals change weekly. When work slips, the manager blames individuals for errors that the setup invites.
Real case: Therac-25 radiation overdoses (1985). Read the full case above.
What it shows for P25. The machine often stopped or delayed treatment, so operators were used to resuming it. Its error messages gave no meaning they could act on. An interface for real operators under real conditions makes a dangerous fault look different from a routine one.
P26. Design for testability¶
The rule: build ways to set and observe the internal state of each part into the design, so that it can be tested
Illustrative, not evidence.
Scenario. A team builds a payment service from several components.
Applying the rule. Each component documents how to set and observe its key states. It has a readiness endpoint, a dry-run mode and a test flag, so the team can test it before release.
Violating the rule. The component can only be tested in production. There is no way to set its state or to see its internal state, so a bug appears only with live customers.
Management variant.
Scenario. A finance team approves access requests for a core system.
Applying the rule. Each approval is stored as a ticket. An auditor can sample the tickets and see who approved what.
Violating the rule. Approvals are given by chat or by phone and leave no record. An auditor cannot sample the control.
P27. Robustness¶
The rule: design so that the result changes little when inputs vary (robustness)
Illustrative, not evidence.
Scenario. A service retries failed calls to a partner API.
Applying the rule. The service uses retries with jitter and idempotent operations. Tests with varied latency, clock drift and message order show stable results.
Violating the rule. The service works only when timing is exact. A small delay in message order makes it create duplicate orders.
Management variant.
Scenario. A team plans a roadmap with a top objective for the year.
Applying the rule. The team checks the plan against a credible range of assumptions. The roadmap still delivers its top objective if one engineer leaves.
Violating the rule. The plan assumes that every engineer stays and every estimate holds. When one engineer leaves, the top objective fails.
Real case: Ariane 5 Flight 501 failure (1996).
On 4 June 1996, about 40 seconds after the start of the flight sequence, the first Ariane 5 veered off its flight path, broke up and exploded. Its guidance used two inertial reference systems (SRIs) in parallel, with identical hardware and software, practically the same as on Ariane 4. Ariane 5's early trajectory gives much higher horizontal velocity than Ariane 4's, and a data conversion that was not protected failed on the larger value. The error came from an alignment function that serves no purpose after lift-off. The back-up SRI failed first, and the active SRI failed 0.05 seconds later for the same reason. The Ariane 5 trajectory data were not in the SRI specification, so the function was never tested under simulated Ariane 5 flight conditions.
What it shows for P27. Ariane 5's early trajectory gave much higher horizontal velocity than Ariane 4's. One unprotected conversion turned that change in input into the loss of both inertial systems. A robust design keeps working when inputs move beyond the range of an earlier use, and is tested with the inputs of the new use.
On 4 June 1996 the maiden flight of the Ariane 5 launcher ended in a failure. Only about 40 seconds after initiation of the flight sequence, at an altitude of about 3700 m, the launcher veered off its flight path, broke up and exploded.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
The design of the Ariane 5 SRI is practically the same as that of an SRI which is presently used on Ariane 4, particularly as regards the software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
There are two SRIs operating in parallel, with identical hardware and software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The data conversion instructions (in Ada code) were not protected from causing an Operand Error, although other conversions of comparable variables in the same place in the code were protected.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
As soon as the launcher lifts off, this function serves no purpose.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The value of BH was much higher than expected because the early part of the trajectory of Ariane 5 differs from that of Ariane 4 and results in considerably higher horizontal velocity values.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
0.05 seconds later the active inertial reference system, identical to the back-up system in hardware and software, failed for the same reason.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
it was jointly agreed not to include the Ariane 5 trajectory data in the SRI requirements and specification.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
Consequently the realignment function was not tested under simulated Ariane 5 flight conditions, and the design error was not discovered.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
the equipment and system tests were not sufficiently representative.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
P28. Accumulated degradation¶
The rule: design for the damage that builds up over the whole design life, not only for the single largest load
Illustrative, not evidence.
Scenario. A team prepares a message broker for production.
Applying the rule. An endurance test runs for several days at a realistic load. The team measures the growth of memory leaks, disk wear and fragmentation.
Violating the rule. The team runs a one-hour peak test and then ships. After three weeks of normal load, the broker runs out of memory.
Management variant.
Scenario. An operations team carries on-call duty.
Applying the rule. The team reviews on-call load and overtime every month. No one works above 48 hours per week as a pattern, and deferred maintenance is tracked.
Violating the rule. The team accepts sustained overtime and lets technical debt grow. Each week looks fine, but burnout and defects build up over the year.
Real case: Patriot missile clock drift at Dhahran (1991).
A Patriot missile battery at Dhahran failed to track and intercept an incoming Scud missile. The Scud hit an Army barracks and killed 28 Americans. A software error made a tracking calculation less accurate the longer the system ran. The battery had run continuously for over 100 hours.
What it shows for P28. The error was small at first and grew with running time, the way wear builds up. The system had not been expected to run continuously for long periods. Design and test for the full time in service, with an endurance run as long as real use, not only for the first hours.
failed to track and intercept an incoming Scud. This Scud subsequently hit an Army barracks, killing 28 Americans.
This problem led to an inaccurate tracking calculation that became worse the longer the system operated.
the battery had been operating continuously for over 100 hours.
nor was it expected to operate continuously for long periods of time.
P29. Compatibility¶
The rule: verify that they are compatible with each other in the real environment, and separate them where they are not
Illustrative, not evidence.
Scenario. A platform team upgrades the container network plugin of a cluster.
Applying the rule. The team keeps a tested compatibility matrix for Kubernetes, CNI and CSI versions. Each interacting pair is on the list before the upgrade.
Violating the rule. The team upgrades the CNI plugin alone. It breaks the storage driver, which was never tested with that version.
Management variant.
Scenario. Two platform teams plan to merge.
Applying the rule. Before the merge, the teams compare their on-call rules and release processes. They plan how to integrate the differences.
Violating the rule. The teams merge on a fixed date without any comparison. Their incompatible release processes block each other for months.
Real case: Apollo 13 oxygen tank heater switches (1970).
The Apollo 13 mission was aborted after oxygen tank no. 2 rapidly lost its oxygen. The tank's heater switches were rated for the spacecraft's 28 V dc supply, but could not open properly at 65 V dc. When Beech ordered switches for the Block II tanks, it did not change their specification to suit 65 V dc. In tests after the accident, the switches failed to open on a 65 V dc supply like the one used at KSC during detanking. No review of documentation and no test had caught the mismatch with the ground support equipment.
What it shows for P29. The switches were compatible with the spacecraft's power, but not with the ground equipment they also met. No review and no test caught the mismatch. Check every pair of parts that interact, in every environment they meet, including ground support.
The Apollo 13 mission was aborted as the direct result of the rapid loss of oxygen from oxygen tank no. 2
the thermostatic switches used, while rated as satisfactory for the 28 V dc spacecraft power supply, could not open properly at 65 V dc.
Beech ordered switches for the Block II tanks but did not change the switch specifications to be compatible with 65 V dc.
The thermostatic switch discrepancy was not detected by NASA, NR, or Beech in their review of documentation, nor did tests identify the incompatibility of the switches with the ground support equipment (GSE) at KSC
it was found that the switches failed to open when the heaters were powered from a 65 V dc supply similar to the power used at KSC during the detanking sequence.
P30. Exact constraint¶
The rule: constrain each property exactly once, with one owner (exact constraint)
Illustrative, not evidence.
Scenario. A deployment is managed by a GitOps tool, and an autoscaler is also enabled.
Applying the rule. The autoscaler owns the replica count, and the GitOps manifest leaves that field out. Each managed field has exactly one owner.
Violating the rule. The GitOps tool and the autoscaler both set the replica count. Each keeps reverting the other, and the system flaps.
Management variant.
Scenario. A team has one backlog and runs incident response.
Applying the rule. There is one product owner per backlog and one incident commander per incident. Each person reports to one person for a given task.
Violating the rule. Two managers set priorities for the same backlog and give opposite orders. The team does not know whom to follow.
P31. Design by selection¶
The rule: generate many variants, test them, keep the best, and repeat (design by selection)
Illustrative, not evidence.
Scenario. A team must tune autoscaler settings, and no model predicts good values.
Applying the rule. The team generates several variants and tests each one under load. It records which variants it kept and repeats the round.
Violating the rule. The team picks the settings by opinion in a meeting, although a test could compare them.
Management variant.
Scenario. A portal team wants to improve developer onboarding.
Applying the rule. The team A/B-tests two onboarding flows with real users and keeps the better one. It then tests the next change.
Violating the rule. The team picks the flow that the loudest person prefers and never tests it with users.
P32. End-to-end¶
The rule: implement it at the end points; add it in lower layers only to improve performance
Illustrative, not evidence.
Scenario. Two services exchange files and need to know that the files arrive intact.
Applying the rule. The sender adds a checksum, and the receiver verifies it in the application. Services authenticate each other with mTLS and do not rely only on network encryption.
Violating the rule. The team trusts the network to deliver data correctly. The application does not check the data, and a corrupted backup is found only at restore time.
Management variant.
Scenario. A platform team supports product teams.
Applying the rule. The platform team measures time-to-first-deploy for product teams. It does not rely only on its own ticket times.
Violating the rule. Each department reports that its tickets close on time. Nobody checks whether a product team can deploy.
P33. Strict interfaces¶
The rule: send strictly according to the specification, reject and report input that violates it instead of silently accepting it, and state units, time zone and reference frames explicitly in every interface
Illustrative, not evidence.
Scenario. A service receives JSON messages from other teams.
Applying the rule. Every interface has a machine-checked schema in CI. The service rejects and logs unknown or malformed fields, and field names carry units, such as latency_ms. Timestamps are in UTC with offset.
Violating the rule. The service silently ignores unknown or malformed fields. A sender writes a duration in seconds, and a receiver reads it as milliseconds.
Management variant.
Scenario. A shared service team receives work requests from other units.
Applying the rule. The team states what it needs in a definition of ready. It returns a request without a cost centre instead of guessing.
Violating the rule. The team silently fixes incomplete requests. Over time, people send incomplete requests as the unofficial standard.
Real case: Mars Climate Orbiter units mismatch (1999).
The Mars Climate Orbiter was lost during its Mars Orbit Insertion maneuver. Ground software produced thruster data in English units, while the interface specification required metric units. The trajectory modelers assumed the data were metric.
What it shows for P33. The interface specification required metric units, but the file held pound-seconds, and the receiving side assumed it followed the specification. A strict interface states its units and checks incoming data against the specification, rejecting data that does not match.
was lost sometime following the spacecraft's entry into Mars occultation during the Mars Orbit Insertion (MOI) maneuver.
Source: NASA: Mars Climate Orbiter Mishap Investigation Board, Phase I Report (1999)
the root cause for the loss of the MCO spacecraft was the failure to use metric units in the coding of a ground software file
Source: NASA: Mars Climate Orbiter Mishap Investigation Board, Phase I Report (1999)
Instead, the data was reported in English units of pound-seconds (lbf-s).
Source: NASA: Mars Climate Orbiter Mishap Investigation Board, Phase I Report (1999)
The data in the AMD file was required to be in metric units per existing software interface documentation, and the trajectory modelers assumed the data was provided in metric units per the requirements.
Source: NASA: Mars Climate Orbiter Mishap Investigation Board, Phase I Report (1999)
P34. Weakest-link scaling¶
The rule: expect rare failures to become frequent, and design and test for the target scale
Illustrative, not evidence.
Scenario. A team tests a storage design on 3 nodes and plans to run it on 300.
Applying the rule. The team estimates the failure rate per component and multiplies it by the planned scale. It then tests the design at the target scale.
Violating the rule. The team assumes that a failure never seen in the 3-node test will not happen at 300 nodes. At scale, the rare failure happens every week.
Management variant.
Scenario. A company grows from 50 to 500 people.
Applying the rule. The company expects rare events, such as fraud or the loss of a key person, to happen regularly. It scales its controls with its size.
Violating the rule. The company keeps the controls of a 50-person firm. At 500 people, an event that happens once in 500 person-years now happens about every year, and no control catches it.
P35. Structure follows the intended system¶
The rule: design team boundaries and communication paths to match the component structure you want, and change them when the intended structure changes
Illustrative, not evidence.
Scenario. A company builds a system from several components and has several teams.
Applying the rule. Each major component boundary has one owning team. The company counts cross-team dependencies and reshapes the teams when the intended structure changes.
Violating the rule. One component is maintained by four teams that never talk to each other. The design then copies their communication gaps.