F. Verification and review¶
How to prove that the work is correct.
Take the Verification and review quiz when you have read the examples.
P54. Verification and validation¶
The rule: do verification AND validation, and before handover test that all parts work together against the owner's requirements (commissioning)
Illustrative, not evidence.
Scenario. A team builds a customer billing portal. Its CI run is green, and the finance owner is waiting for it.
Applying the rule. The CI results are on record (verification). The finance owner has signed that the portal meets their requirements (validation). Before handover, the team runs one end-to-end test of DNS, certificates, monitoring and backup together. Both a test result and a stakeholder acceptance exist.
Violating the rule. The team marks the portal done because CI is green. Nobody asks the finance owner to accept it, and nobody tests the parts together. On launch day the certificate service rejects the new DNS name.
Management variant.
Scenario. A company launches a developer portal to cut the time new teams spend on setup.
Applying the rule. The portal is accepted against its specification (verification). Six months later the company measures adoption and checks that the business case still holds (validation).
Violating the rule. The portal passes its acceptance checklist and the project closes. Nobody measures whether teams use it, and the business case is never revisited.
Real case: Opening of Heathrow Terminal 5 (2008).
Heathrow Terminal 5 opened in 2008 with serious problems, and 23,205 bags needed manual sorting. The House of Commons Transport Committee found two main causes: poor communication between owner and operator, and poor staff training and system testing. British Airways said it had compromised the testing to absorb building delays. The public trials did not get the system to work well enough for the opening.
What it shows for P54. The terminal opened before its systems and staff had been tested well enough together. British Airways said it compromised on the testing regime because the building work was late, and the proving trials did not get the system to the point where it could cope with the opening. Commissioning tests that all parts work together against the owner's requirements before handover.
According to British Airways, 23,205 bags required manual sorting before being returned to their owners.
Source: House of Commons Transport Committee: The opening of Heathrow Terminal 5 (HC 543, 2008)
We found that most of these problems were caused by one of two main factors: insufficient communication between owner and operator, and poor staff training and system testing.
Source: House of Commons Transport Committee: The opening of Heathrow Terminal 5 (HC 543, 2008)
In particular, we had compromised on the testing regime as result of delays in completing the building programme for T5
Source: House of Commons Transport Committee: The opening of Heathrow Terminal 5 (HC 543, 2008)
The proving trials may have succeeded in identifying improvements and enabling members of the public to experience the new terminal but they failed in the ultimate objective of getting the system to a point where it worked well enough to cope with the opening successfully.
Source: House of Commons Transport Committee: The opening of Heathrow Terminal 5 (HC 543, 2008)
P55. Find defects early¶
The rule: find and fix defects at the earliest possible stage
Illustrative, not evidence.
Scenario. A team adds a feature to a service that many other teams call.
Applying the rule. Lint, type checks and unit tests run on every pull request before merge. Each check runs at the first stage where it can run, so a type error stops the change before review.
Violating the rule. Lint and tests run nightly after the release branch is cut. A type error reaches production and is found at 2 am by a customer.
Management variant.
Scenario. A product lead wants to start a quarter of work for twelve people on a new initiative.
Applying the rule. A one-page proposal and its business case are reviewed before any people or money are committed. The review finds a missing dependency, and the plan changes before work starts.
Violating the rule. The lead commits the team first and reviews the business case at the end of the quarter, when the money is already spent.
P56. Independent checker¶
The rule: have a checker review the work, and raise the checker's independence with the consequence of failure: self → peer → external organization
Illustrative, not evidence.
Scenario. A payments team changes the code that authorizes card transactions.
Applying the rule. A second engineer approves every pull request, and the author never merges their own change. Because a failure here is costly, the team also commissions an external penetration test. The checker's independence matches the rigor matrix.
Violating the rule. The author reads their change twice, approves it and merges it. Nobody else sees it before release.
Management variant.
Scenario. A company checks its internal budgets and its yearly accounts.
Applying the rule. Finance checks a team lead's budget. An external auditor checks the company's accounts, and the firm rotates its long-serving external auditors.
Violating the rule. The head of a department prepares the budget and approves it. The same audit firm has checked the accounts for twenty years.
Real case: Clapham Junction rail crash (1988).
On 12 December 1988, a crowded commuter train ran into the rear of a stationary train near Clapham Junction, and 35 people died. During signalling work, an old wire was left connected at the fuse end and loose at the relay end. It made contact at a relay and gave a false feed of current that by-passed another relay, so signal WF138 showed a proceed aspect instead of red. The wiring errors were not found by the installer, his supervision or the testing before the signal was put into use. The independent wire count that practice and instructions called for was not done.
What it shows for P56. Neither the installer, nor his supervision, nor the testing found his errors. The report says such rare errors have to be guarded against by independent checking of his work, and the independent wire count that good practice called for was not done.
a crowded commuter train ran head-on into the rear of another which was stationary in a cutting just south of Clapham Junction station.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
As a result of the accident 35 people died and nearly 500 were injured, 69 of them seriously.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
signal WF138 would fail to turn to red and would instead show a proceed aspect.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
It would be a failure which falls on the wrong side of safety.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
At the fuse end even more surprisingly he did not disconnect the old wire at all.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
Although disconnected at the relay end the wire was left loose and made metal-to-metal contact at DM TRR, a false feed of current to by-pass DL TRR.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
it is actually an independent wire count that both good practice and Departmental Instructions called for.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
Those errors and their potential for disaster were not discovered by Mr Hemingway himself, were not discovered by any supervision of him, and were not discovered in the testing process before the signal was commissioned that evening and the line put back into public use next morning.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
It is those unusual and infrequent events that have to be guarded against by a system of independent checking of his work.
Source: Hidden: Investigation into the Clapham Junction Railway Accident (1989)
P57. Test as you operate¶
The rule: run the test in the same configuration and conditions as real operation, and test again after every repair
Illustrative, not evidence.
Scenario. A team must show that a new release of its web service is ready.
Applying the rule. Staging runs the same image digest and Helm values as production. The team lists the few differences, such as a smaller database, and justifies each one. After fixing a failed test, it runs the test again.
Violating the rule. Staging runs an image built separately from the release, with TLS switched off. The test passes, and the different build is deployed. After a fix, the team ships without running the test again.
Management variant.
Scenario. A company plans a new on-call model for its engineering teams.
Applying the rule. One team pilots the model for one month under real conditions before full rollout. The company exercises its continuity plan once a year.
Violating the rule. The company announces the new on-call model to all teams at once. The continuity plan has never been exercised.
Real case: Ariane 5 Flight 501 failure (1996).
On 4 June 1996, about 40 seconds after the start of the flight sequence, the first Ariane 5 veered off its flight path, broke up and exploded. Its guidance used two inertial reference systems (SRIs) in parallel, with identical hardware and software, practically the same as on Ariane 4. Ariane 5's early trajectory gives much higher horizontal velocity than Ariane 4's, and a data conversion that was not protected failed on the larger value. The error came from an alignment function that serves no purpose after lift-off. The back-up SRI failed first, and the active SRI failed 0.05 seconds later for the same reason. The Ariane 5 trajectory data were not in the SRI specification, so the function was never tested under simulated Ariane 5 flight conditions.
What it shows for P57. The Ariane 5 trajectory data were left out of the SRI specification, so the realignment function was never tested under simulated Ariane 5 flight conditions, and the design error was not found. A readiness test runs in the conditions of real operation, and each difference from them is listed and justified.
On 4 June 1996 the maiden flight of the Ariane 5 launcher ended in a failure. Only about 40 seconds after initiation of the flight sequence, at an altitude of about 3700 m, the launcher veered off its flight path, broke up and exploded.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
The design of the Ariane 5 SRI is practically the same as that of an SRI which is presently used on Ariane 4, particularly as regards the software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
There are two SRIs operating in parallel, with identical hardware and software.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The data conversion instructions (in Ada code) were not protected from causing an Operand Error, although other conversions of comparable variables in the same place in the code were protected.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
As soon as the launcher lifts off, this function serves no purpose.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
The value of BH was much higher than expected because the early part of the trajectory of Ariane 5 differs from that of Ariane 4 and results in considerably higher horizontal velocity values.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
0.05 seconds later the active inertial reference system, identical to the back-up system in hardware and software, failed for the same reason.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
it was jointly agreed not to include the Ariane 5 trajectory data in the SRI requirements and specification.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
Consequently the realignment function was not tested under simulated Ariane 5 flight conditions, and the design error was not discovered.
Source: ESA and CNES: Ariane 5 Flight 501 Failure, Report by the Inquiry Board (1996)
the equipment and system tests were not sufficiently representative.
Source: ESA: Ariane 501 - Presentation of Inquiry Board report (1996)
P58. Comparable measurement¶
The rule: measure under stated, controlled reference conditions, report the uncertainty with every result, and calibrate instruments regularly against traceable references
Illustrative, not evidence.
Scenario. A team claims that a new cache makes its service faster.
Applying the rule. Both versions run on the same instance type with a fixed load and a warm-up period. The report states these reference conditions and the spread over five runs. A reader can see whether the difference is larger than the spread.
Violating the rule. The team compares one run on a laptop with one run on a CI runner. The report gives a single number for each and no spread.
Management variant.
Scenario. Two directors want to compare lead time across teams.
Applying the rule. Lead time is defined once, with its formula, period, population and data source. The report shows how much lead time normally varies before it compares teams.
Violating the rule. One team measures lead time from ticket creation and another from the first commit. The directors rank the teams on these numbers.
Real case: Hubble Space Telescope primary mirror (1990).
After its launch on 24 April 1990, the Hubble Space Telescope could not be properly focused. Its primary mirror had been made in the wrong shape, with an error ten times the specified tolerance. The mirror was shaped and verified with one instrument, the reflective null corrector, whose dimensions were not verified after assembly. Other tests that disagreed with it were dismissed as less precise.
What it shows for P58. The mirror was made and verified with one instrument, whose dimensions were not verified after assembly. When other instruments gave different results, those results were dismissed as less precise. Instruments calibrated against a traceable reference, with the uncertainty of each result stated, make a disagreement a reason to investigate rather than to dismiss.
The Hubble Space Telescope (HST) was launched aboard the Space Shuttle Discovery on April 24, 1990.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
During checkout on orbit, it was discovered that the telescope could not be properly focused because of a flaw in the optics.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
The error is ten times larger than the specified tolerance.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
The Perkin-Elmer plan for fabricating the primary mirror placed complete reliance on the reflective null corrector as the only test to be used in both manufacturing and verifying the mirror's surface with the required precision.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
No verification of the reflective null corrector's dimensions was carried out by Perkin-Elmer after the original assembly.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
This information was dismissed, as it was in the case for the INC, because the RvNC was believed to be less precise than the RNC and therefore not reliable.
Source: NASA: The Hubble Space Telescope Optical Systems Failure Report (1990)
P59. Traceability¶
The rule: link every requirement in both directions to design and test
Illustrative, not evidence.
Scenario. A company writes firmware for an infusion pump, which is safety-critical and regulated.
Applying the rule. Each requirement links to its design element and to a test ID in the ticket system, and each test links back to a requirement. The traceability report shows no requirement without a test and no test without a requirement.
Violating the rule. The report shows three requirements with no test and five tests that link to no requirement. The team points to its 97 percent line coverage.
Management variant.
Scenario. A company must show that it meets its data protection obligations.
Applying the rule. Each obligation in the register points to an owner, a control and an audit test of that control.
Violating the rule. The register lists the obligations, but several have no control. One control has never been tested.
P60. Codes as minimum¶
The rule: meet it as the minimum, then apply the other rules
Illustrative, not evidence.
Scenario. A team runs a Kubernetes cluster, and a published benchmark (the CIS Benchmark for Kubernetes) applies to it.
Applying the rule. The team runs the compliance scan, and it passes. The team treats this as the minimum and then applies the other rules, such as failure analysis of its own design.
Violating the rule. The scan passes, and the team lead says the cluster is secure and no further analysis is needed.
Management variant.
Scenario. A company must follow working-time law for its on-call staff.
Applying the rule. The company meets the law as the floor. It then applies the other rules, such as psychological safety (P5) and accumulated degradation (P28), which go further.
Violating the rule. The company schedules on-call shifts to the legal limit and treats that as proof that the schedule is fair and sustainable.
Real case: Hyatt Regency walkway collapse (1981).
On 17 July 1981, two suspended walkways in the atrium of the Hyatt Regency Hotel in Kansas City collapsed. The National Bureau of Standards found that the box beam-hanger rod connections did not meet the Kansas City Building Code, even in the original design. A change of the hanger rods during construction about doubled the load on the fourth floor connections. The walkways fell under loads well below the design loads of the code.
What it shows for P60. The code sets the minimum capacity. The connections did not meet it as built, and would not have met it in the original design either. Meeting the code was the floor, and here even the floor was missing.
On July 17, 1981, two suspended walkways within the atrium area of the Hyatt Regency Hotel in Kansas City, Mo., collapsed, leaving 113 people dead and 186 injured.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
Two factors contributed to the collapse: inadequacy of the original design for the box beam-hanger rod connection, which was identical for all three walkways, and a change in hanger rod arrangement during construction that essentially doubled the load on the box beam-hanger rod connections at the fourth floor walkway.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
the maximum load on a fourth floor box beam-hanger rod connection at the time of collapse was only 31 percent of the ultimate capacity expected of a connection designed under the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
they had only minimal capacity to resist their own weight and had virtually no capacity to resist additional loads imposed by people.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
Collapse of the walkways occurred under the action of loads that were substantially less than the design loads specified by the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
As constructed, the box beam-hanger rod connections, the fourth floor to ceiling hanger rods, and the third floor walkway hanger rods did not satisfy the design provisions of the Kansas City Building Code.
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
The box beam-hanger rod connection would not have satisfied the Kansas City Building Code under the original hanger rod detail (continuous rod)
Source: NBS BSS 143: Investigation of the Kansas City Hyatt Regency Walkways Collapse (1982)
P61. Qualified procedure and person¶
The rule: prove the procedure by a test before using it, and qualify each person who performs it
Illustrative, not evidence.
Scenario. A team depends on database backups, and a restore is the only way to know that a backup works.
Applying the rule. The restore procedure is proven by a real restore, and the test record is filed. Only engineers who have completed a restore drill run production restores, and each has a qualification record.
Violating the rule. The backup procedure was written in 2022 and has never been restored. Any on-call engineer may run it during an outage.
Management variant.
Scenario. A company wants consistent hiring interviews, and the quality of an interview cannot be checked afterwards.
Applying the rule. The interview procedure is proven by a pilot. Only interviewers who have completed the training run structured interviews (P62).
Violating the rule. Any manager runs interviews in their own way. Nobody has tested the procedure or trained the interviewers.
P62. Structured selection of people¶
The rule: use structured, job-related interviews (the same questions for each candidate, answers rated on scales anchored to job requirements), and add work samples or job-knowledge tests
Illustrative, not evidence.
Scenario. A company hires a platform engineer.
Applying the rule. Every candidate answers the same six job-related questions. Two raters score the answers on scales anchored to job requirements, each before the discussion. Every candidate also does the same debugging exercise.
Violating the rule. Each interviewer holds a 'culture fit' chat with different questions for each candidate. The raters discuss first and score afterwards.
P63. Feedback on the task¶
The rule: focus the feedback on the task and on how to improve it, not on the person's character or ranking, and check afterwards whether performance improved
Illustrative, not evidence.
Scenario. An engineer deploys a service, and the runbook is missing the rollback step.
Applying the rule. The lead says, "The runbook is missing the rollback step. Add it and test it." The feedback record names the task and the improvement. At the next review the lead checks whether the runbook now has a tested rollback step.
Violating the rule. The lead says, "You are careless," and ranks the engineer last on the team. The record names no task and no improvement, and nobody checks later.