B. Knowledge and decisions¶
How requirements, decisions, models and evidence are recorded and trusted.
Take the Knowledge and decisions quiz when you have read the examples.
P9. Measurable requirements¶
The rule: write each requirement as measurable, and link it to a user need
Illustrative, not evidence.
Scenario. A platform team plans a faster login service for an internal portal.
Applying the rule. The requirement reads "p95 login time is under 300 ms at 500 concurrent users", with a test that checks it. It links to ticket UX-212, where staff report slow sign-ins. Every requirement in the spec has a criterion and a link to a need, so the check passes.
Violating the rule. The spec says "the login shall be fast". Nobody can test it, and no ticket says who needs it. The team cannot show that the service meets the requirement.
Management variant.
Scenario. A director sets a goal for the cloud platform group for the year.
Applying the rule. The objective reads "cut median time to provision a namespace from 3 days to 1 hour by the end of Q2". It has one named owner and links to the product teams' request for self-service. Measure, target, deadline, owner and need are all present.
Violating the rule. The objective reads "make provisioning better". It has no measure, no deadline and no owner, and it does not say whose need it serves.
P10. Decision records¶
The rule: record the decision, why you made it, and which alternatives you rejected
Illustrative, not evidence.
Scenario. A team chooses a message queue for a new order service. The choice will be hard to change later.
Applying the rule. The team writes a decision record in the repository. It states the choice, the reason (ordering guarantees and the team's operating experience) and the two rejected alternatives with why each lost. Two years later a new engineer can judge whether the reason still holds.
Violating the rule. The team picks the queue in a call and writes nothing down. Two years later nobody knows why the system is built this way, and no one dares to change it.
Management variant.
Scenario. A department head reorganizes on-call duty across three teams.
Applying the rule. The decision log gets an entry. It names the shared rotation as the choice, gives the reason (fewer night calls per person), lists the per-team rotation as rejected with its reason, and names the owner.
Violating the rule. The new rotation is announced in a chat message. The reason, the alternatives and the owner are not recorded, so the next manager cannot tell why it was chosen.
P11. Trustworthy evidence¶
The rule: make it attributable, legible, recorded at the time, original and accurate; changes must keep the original entry visible and record who, when and why
Illustrative, not evidence.
Scenario. A release depends on a test report that a manager must approve.
Applying the rule. The report is stored with the commit SHA and signed by the person who ran it. A correction is a new entry that keeps the original visible and records who changed it, when and why. A sample of records shows author, time and full history.
Violating the rule. After a failed run, someone edits the report so that the test shows as passed. The original entry is gone, and no one can tell who changed it or when.
Management variant.
Scenario. A finance team keeps the project budget in a shared ledger.
Applying the rule. A budget change is a new entry with author, date and reason. The earlier figure stays visible. An auditor can reproduce every past state and see that the records are in order.
Violating the rule. A manager overwrites the budget cell with a new number. The old figure is lost, and the organization cannot show how or why the number changed.
P12. Model validity¶
The rule: validate the model for its intended use, use it only inside the range it was validated for (domain of applicability), and check each result against an independent rough estimate (order of magnitude)
Illustrative, not evidence.
Scenario. A team sizes a cluster for next year's traffic from a benchmark. The benchmark ran on three, four and six nodes.
Applying the rule. The sizing note states that the benchmark is valid for 3 to 6 nodes. The plan targets 5 nodes, inside that range. The team also checks the result against a rough back-of-the-envelope estimate, and both agree within an order of magnitude.
Violating the rule. The team scales the benchmark results up to predict 50 nodes. The benchmark never covered that range, and no independent estimate checks the result. The decision rests on an unvalidated extrapolation.
Management variant.
Scenario. A programme office reviews the business case for a platform migration.
Applying the rule. The office compares the estimate with the last five migrations, which overran by 40 % on average. It adjusts the plan with that outside view instead of trusting only the team's own plan.
Violating the rule. The office accepts the team's own estimate as it stands. It never looks at how similar past projects ended.
Real case: Space Shuttle Columbia foam strike and its organizational causes (2003).
A piece of insulating foam from the External Tank struck Columbia's left wing after launch. During re-entry, the breach it made in the Thermal Protection System let superheated air in, and the orbiter and its crew were lost. During the flight, engineers assessed the strike with Crater, a model calibrated on far smaller debris. Program management cancelled a request for images of the wing. The Columbia Accident Investigation Board did not stop at the part that broke or the people closest to it, and looked for causes in the organization as a whole. It found that the causes of the institutional failure behind Challenger, 17 years earlier, had not been fixed.
What it shows for P12. Crater was calibrated and validated on small pieces of debris. Engineers used it on a piece up to 640 times larger in volume, far outside that range. A model result is only valid inside the range it was validated for.
The physical cause of the loss of Columbia and its crew was a breach in the Thermal Protection System on the leading edge of the left wing.
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 3, Accident Analysis (2003)
The breach was initiated by a piece of insulating foam that separated from the left bipod ramp of the External Tank and struck the wing in the vicinity of the lower half of Reinforced Carbon-Carbon panel 8 at 81.9 seconds after launch.
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 3, Accident Analysis (2003)
During re-entry, this breach in the Thermal Protection System allowed superheated air to penetrate the leading-edge insulation
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 3, Accident Analysis (2003)
When used within its validated limits, Crater provides conservative predictions
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 6, Decision Making at NASA (2003)
engineers used Crater during STS-107 to analyze a piece of debris that was at maximum 640 times larger in volume than the pieces of debris used to calibrate and validate the Crater model
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 6, Decision Making at NASA (2003)
When asked why he did not send this e-mail, Rocha replied that he did not want to jump the chain of command.
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 6, Decision Making at NASA (2003)
this is the wrong (and bordering on irresponsible) answer from the SSP and Orbiter not to request additional imaging help from any outside source.
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 6, Decision Making at NASA (2003)
Even after the imagery request had been cancelled by Program management
Source: Columbia Accident Investigation Board: Report, Volume I, Chapter 6, Decision Making at NASA (2003)
empowered any employee to stop an operation at the mere glimmer of a problem.
Unfortunately, NASAʼs views of its safety culture in those briefings did not reflect reality.
Even after it was clear from the launch videos that foam had struck the Orbiter in a manner never before seen
were largely silent during the events leading up to the loss of Columbia.
Many accident investigations make the same mistake in defining causes. They identify the widget that broke or malfunctioned, then locate the person most closely connected with the technical failure
Such corrections lead to a misguided and potentially disastrous belief that the underlying problem has been solved.
A central piece of our expanded cause model involves NASA as an organizational whole.
That silence was not merely a failure of safety, but a failure of the entire organization.
the causes of the institutional failure responsible for Challenger have not been fixed.
Seventeen years separated the two accidents.
problems cannot be solved simply by retirements, resignations, or transferring personnel.
The Board had a hard time understanding how, after the bitter lessons of Challenger, NASA could have failed to identify a similar trend.
P13. Automated reproducible results¶
The rule: produce it only through automated, recorded steps, with versions of data, code and tools
Illustrative, not evidence.
Scenario. A data team publishes a monthly capacity report to management.
Applying the rule. One command regenerates the report from the recorded inputs. The data snapshot, the code and the tool versions are pinned. Anyone can re-run it and get the same figures.
Violating the rule. An analyst copies numbers from three dashboards into a spreadsheet by hand. The steps are not recorded, so nobody can regenerate the report, and a typo goes unnoticed.
Management variant.
Scenario. A delivery lead reports DORA metrics to the executive team.
Applying the rule. The dashboard is generated from pipeline data by a documented job. Each figure names its data source, and the same job produces the same numbers on any day.
Violating the rule. The figures are typed into a hand-edited spreadsheet each quarter. The sources are not named, and nobody can reproduce last quarter's numbers.
P14. Provenance¶
The rule: trace each part to its origin, batch and certificate (provenance)
Illustrative, not evidence.
Scenario. A team deploys a container image to production.
Applying the rule. The image is signed and carries build provenance and a software bill of materials. The deployment pins the image by digest. For this artifact the team can show its origin, build and certificate. When a library flaw is announced, they find every affected deployment.
Violating the rule. The team runs an unsigned image from an unknown registry. They cannot show where it came from or what it contains. When a defect appears, they cannot find all affected deployments.
Management variant.
Scenario. A procurement team is about to sign with a SaaS vendor.
Applying the rule. Before signing, the team lists the vendor's sub-processors and where each stores data. It acts on what it finds, for example by requiring data to stay in the region.
Violating the rule. The team signs on the strength of a sales presentation. It does not know the vendor's sub-processors or where they hold the data.
P15. Precaution¶
The rule: take proportionate, cost-effective protective measures without waiting for full proof
Illustrative, not evidence.
Scenario. A database team sees odd checksum errors on a storage volume. The cause is unknown.
Applying the rule. The team takes a snapshot and stops writes to the volume at once. It does not wait for the root cause. A record states the protective measure and why it is proportionate to the risk of lasting data loss.
Violating the rule. Writes continue while the team waits for proof of corruption. The damage is irreversible, and by the time the proof arrives, more data is lost.
Management variant.
Scenario. Finance receives a fraud warning about a supplier's new bank account details.
Applying the rule. Payments to the supplier are paused while the warning is checked. The step is proportionate and reversible, and it is taken before fraud is proven.
Violating the rule. Payments continue until the fraud is confirmed. By then the money is gone.
Real case: Space Shuttle Challenger O-ring joint and launch decision (1986).
The Space Shuttle Challenger was lost when the seals in a joint of the right Solid Rocket Motor failed. The joint's O-rings had shown erosion and blow-by on earlier flights, and NASA and its contractor, Morton Thiokol, came to accept this as an acceptable flight risk instead of solving it. For this launch, the contractor's engineers had first recommended against launch below 53 degrees Fahrenheit. Their management reversed that position at the urging of NASA's Marshall Space Flight Center. The Rogers Commission found the decision to launch flawed, and found that it was based on incomplete and sometimes misleading information.
What it shows for P15. The harm was catastrophic and the evidence on cold seals was incomplete. One engineer testified that the meeting put them in the position of proving that launch was unsafe, instead of showing there was enough data to launch. Precaution turns that burden around: when the harm is irreversible, act on the warning without waiting for full proof.
the loss of the Space Shuttle Challenger was caused by a failure in the joint between the two lower segments of the right Solid Rocket Motor.
The decision to launch the Challenger was flawed.
the initial written recommendation of the contractor advising against the launch at temperatures below 53 degrees Fahrenheit
a decision to launch 51-L based on incomplete and sometimes misleading information
The waiving of launch constraints appears to have been at the expense of flight safety.
the Thiokol Management reversed its position and recommended the launch of 51-L, at the urging of Marshall and contrary to the views of its engineers in order to accommodate a major customer.
we were being put in a position to prove that we should not launch rather than being put in the position and prove that we had enough data to launch.
The genesis of the Challenger accident-the failure of the joint of the right Solid Rocket Motor-began with decisions made in the design of the joint
Morton Thiokol, Inc., the contractor
as tests and then flights confirmed damage to the sealing rings, the reaction by both NASA and Thiokol was to increase the amount of damage considered “acceptable.”
Neither organization developed a solution to the unexpected occurrences of O-ring erosion and blow-by even though this problem was experienced frequently during the Shuttle flight history. Instead, Thiokol and NASA management came to accept erosion and blow-by as unavoidable and an acceptable flight risk.
NASA and Thiokol accepted escalating risk apparently because they “got away with it last time.”
Tracking and continuing only anomalies that are “outside the data base” of prior flight allowed major problems to be removed from, and lost by, the reporting system.
P16. Set-based design¶
The rule: develop several alternatives in parallel, eliminate weak ones as evidence arrives, and commit only after feasibility is shown (set-based design; opposite: point-based design)
Illustrative, not evidence.
Scenario. A team must pick an ingress controller. No one knows which will meet the latency goal.
Applying the rule. The team prototypes two controllers in parallel. Load tests eliminate the weaker one. The design record lists both alternatives and the evidence that removed the loser. The team commits only after feasibility is shown.
Violating the rule. The team chooses one controller on day one. Late in the project, it finds that the controller cannot meet the latency requirement. The rework is expensive.
Management variant.
Scenario. A company needs a new monitoring vendor.
Applying the rule. Two vendors run a paid pilot for a quarter. The weaker one is dropped as results arrive, and only then is one contract signed.
Violating the rule. Leadership signs a three-year contract with the first vendor it meets. It learns after launch that the tool does not fit.
P17. Technology maturity¶
The rule: assess how mature the technology is, and keep immature technology out of critical paths
Illustrative, not evidence.
Scenario. A team wants a young open-source project for the control plane of its platform.
Applying the rule. The team records a maturity assessment for the project: it is early stage and has few production users. It keeps the project off the critical path and uses a graduated, proven project there instead.
Violating the rule. The team adopts an untested tool for the production control plane with no assessment. Unknown failures now sit on the critical path.
Management variant.
Scenario. A company considers a start-up vendor for its payroll process.
Applying the rule. The company assesses the vendor's maturity first. It does not choose the start-up for payroll without an exit plan.
Violating the rule. The company moves payroll to a two-person start-up without checking its maturity or planning an exit. If the vendor fails, payroll stops.
P18. Decision speed by reversibility¶
The rule: classify it as reversible (two-way door) or irreversible (one-way door); make reversible decisions quickly at the lowest competent level, and irreversible decisions slowly, with consultation and review
Illustrative, not evidence.
Scenario. A team has two decisions this week: trying a new linter rule, and moving the whole service to another cloud provider.
Applying the rule. The team classifies the linter rule as reversible and decides it today at team level. It classifies the provider move as irreversible. That decision goes through consultation, a red team and a decision record (P10). Each record states its class and the matching process.
Violating the rule. The linter trial waits three weeks for committee approval. The provider contract is signed in a single meeting. The process is heavy where it is not needed and light where harm lasts.
P19. Indicators are not targets¶
The rule: expect it to be gamed: pair it with a counter-measure, compare it with direct observation, and change it when it drifts from the real objective
Illustrative, not evidence.
Scenario. A manager wants to rank on-call engineers by a single number.
Applying the rule. The manager pairs the closed ticket count with a counter-indicator, the number of reopened incidents, and reads a sample of incidents to compare the numbers with what really happened. Each year the team reviews whether both still track on-call quality. Mean time to restore is not a bonus target.
Violating the rule. Closed ticket count is the only measure of on-call quality. Engineers split tickets and close them early to raise the number, and real quality drops.
Real case: Wells Fargo sales goals and unauthorized accounts (2016).
Wells Fargo set sales goals and paid incentives to raise the number of products its employees sold. Thousands of employees opened accounts and credit card applications without customers' knowledge or consent to meet the goals and earn rewards. Senior leaders kept the goals even when regional leaders challenged them as too high.
What it shows for P19. The bank rewarded the count of products sold, so employees inflated the count with accounts nobody wanted. Some even moved customers' money to make new accounts look funded. A rewarded indicator needs a counter-indicator, such as account quality, and a check against what customers really asked for.
Respondent set sales goals and implemented sales incentives, including an incentive-compensation program, in part to increase the number of banking products and services that its employees sold to its customers.
Source: CFPB: Consent Order, In the Matter of Wells Fargo Bank, N.A., 2016-CFPB-0015 (2016)
Thousands of Respondent's employees engaged in Improper Sales Practices to satisfy sales goals and earn financial rewards
Source: CFPB: Consent Order, In the Matter of Wells Fargo Bank, N.A., 2016-CFPB-0015 (2016)
opened unauthorized deposit accounts for existing customers and transferred funds to those accounts from their owners' other accounts, all without their customers' knowledge or consent
Source: CFPB: Consent Order, In the Matter of Wells Fargo Bank, N.A., 2016-CFPB-0015 (2016)
To qualify for incentives that rewarded bankers for opening new accounts that were funded shortly after opening, Respondent's employees opened deposit accounts without consumers' knowledge or consent
Source: CFPB: Consent Order, In the Matter of Wells Fargo Bank, N.A., 2016-CFPB-0015 (2016)
Even when challenged by their regional leaders, the senior leadership of the Community Bank failed to appreciate or accept that their sales goals were too high and becoming increasingly untenable.
They also failed to adequately consider that low quality accounts could be indicative of unauthorized accounts.
P20. Stop criteria set in advance¶
The rule: write down, before starting, the evidence and the decision points (gates) at which it stops, and let someone who did not start it make the continue-or-stop decision
Illustrative, not evidence.
Scenario. A sponsor launches a service-mesh pilot with an uncertain outcome.
Applying the rule. Before the start, the team writes the stop criteria: stop if p99 latency rises by more than 5 ms after 4 weeks. The gate dates are set. The architecture board, not the sponsor, makes the continue-or-stop decision.
Violating the rule. The pilot has no stop criteria. After poor results, the sponsor says "we have invested too much to stop now" and adds more people. The sponsor also makes the decision alone.
P21. Specific, challenging goals with feedback¶
The rule: set few, specific and challenging goals, give regular feedback on progress toward them, and for new or complex tasks set learning goals first
Illustrative, not evidence.
Scenario. A lead sets a goal for a platform team this quarter.
Applying the rule. The goal reads "cut median build time from 12 to 6 minutes this quarter". A weekly chart shows progress. For a new technology the goal is to run three spikes and document what the team learned. Each goal has a measure, a deadline and a progress chart.
Violating the rule. The only goal is "do your best". Nobody shows progress, so the team cannot tell whether it is on track.