G. Change¶
How to change the system safely.
Take the Change quiz when you have read the examples.
P64. Versioning and change review¶
The rule: version every artifact; before each change, review its technical basis, its safety impact and the procedures it affects, approve it, and inform the affected people before go-live
Illustrative, not evidence.
Scenario. A small team runs a billing service. A developer wants to raise a database connection limit.
Applying the rule. The developer opens a pull request that states why the limit changes, what load and failure effects it has, and which runbook it touches. A second engineer approves it. The on-call team is told before go-live. The pipeline deploys the approved commit, and the deployed state matches it.
Violating the rule. The developer logs in to production and edits the setting by hand. Nobody reviews it and nobody tells the on-call team. The running service now differs from the repository, so the next failure is hard to reproduce.
Management variant.
Scenario. A company keeps its on-call policy in a shared document that anyone can edit.
Applying the rule. The policy is versioned in git. Each change is a pull request that names the reason, lists the teams affected and is approved. The affected people are told before the new version takes effect.
Violating the rule. A manager edits the shared document the evening before a holiday. Nobody reviews the change, no version is kept, and the on-call people learn about it when the pager goes off.
Real case: Knight Capital trading loss (2012).
Knight Capital deployed new trading code to its order router, SMARS. The code reused a flag once used by an old function, Power Peg. Knight had stopped using Power Peg years earlier, but its code was still present and callable. A technician did not copy the new code to one of the eight servers, and no second technician reviewed the deployment. On 1 August 2012, in about 45 minutes, the old code on that server sent millions of orders into the market, while staff tried to find the source of the problem. Before the market opened, an internal system had sent 97 e-mails that pointed at the fault, but they were not designed as alerts. Knight lost 460 million dollars.
What it shows for P64. The deployed state did not match the intended change. One of eight servers kept the old code, and no second technician reviewed the deployment. P64's check, that the deployed state matches an approved change, is aimed at exactly this gap.
On August 1, 2012, Knight Capital Americas LLC (“Knight”) experienced a significant error in the operation of its automated routing system for equity orders, known as SMARS.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
This unused code previously had been used for functionality called “Power Peg,” which Knight had discontinued using many years earlier.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Despite the lack of use, the Power Peg functionality remained present and callable at the time of the RLP deployment.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
The new RLP code also repurposed a flag that was formerly used to activate the Power Peg code.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Ultimately, Knight realized a $460 million loss on these positions.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
SMARS routed millions of orders into the market over a 45-minute period
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system sent 97 of these e-mail messages to a group of Knight personnel before the 9:30 a.m. market open.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system continued to send millions of child orders while its personnel attempted to identify the source of the problem.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight needed clear guidance for its technology personnel as to when to disconnect a malfunctioning system from the market.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
P65. Small verified steps¶
The rule: make the change in small steps, check the result of each step before the next one, and define each check as a hold point (work stops until the check is approved) or a witness point (work continues while the check is observed)
Illustrative, not evidence.
Scenario. A team ships a new checkout service to replace the old one.
Applying the rule. The team defines the steps 1 %, 10 % and 100 % of traffic. It checks error rate and latency after each step before it starts the next. The check before 100 % is a hold point, so work stops until a lead approves it. The earlier checks are witness points, so the rollout continues while an engineer observes them.
Violating the rule. The team switches all traffic to the new service at once. A defect hits every customer at the same time, and there is no earlier check that could have limited the harm.
Management variant.
Scenario. A company plans to change how all its offices book travel.
Applying the rule. The company pilots the new process in one department. At the end of the pilot, a named group decides whether to continue, change the plan or stop. Only then does the next department start.
Violating the rule. The company switches every office to the new process on the same day. It decides nothing at any stage, so a flaw reaches all offices before anyone can stop it.
Real case: Cloudflare global outage from a firewall rule (2019) (2019).
On 2 July 2019 Cloudflare deployed a new rule to its web application firewall. The rule exhausted the CPUs that serve HTTP and HTTPS traffic across its whole network, and visitors saw 502 errors. The procedure for rule changes allowed a global push without a staged rollout, so the change went global in seconds. Synthetic tests paged the team three minutes later, but the alert for the global traffic drop was slow.
What it shows for P65. Software at Cloudflare went out in stages, but firewall rules went to every site at once. Cloudflare's remedy is to roll out rules in stages too, and keep a global push only for emergencies.
On July 2, we deployed a new rule in our WAF Managed Rules that caused CPUs to become exhausted on every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
This resulted in our customers (and their customers) seeing a 502 error page when visiting any Cloudflare domain.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
The SOP allowed a non-emergency rule change to go globally into production without a staged rollout.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
However, in this case, that speed meant that a change to the rules went global in seconds.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
Three minutes later the first PagerDuty page went out indicating a fault with the WAF. This was a synthetic test that checks the functionality of the WAF (we have hundreds of such tests) from outside Cloudflare to ensure that it is working correctly.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
The first alert for the global traffic drop took too long to fire.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
P66. Isolation before work¶
The rule: isolate that part from all inputs and energy, and lock the isolation so that nobody can restart it unexpectedly
Illustrative, not evidence.
Scenario. An operations team needs to replace a disk in a cluster node.
Applying the rule. The procedure starts with an isolation step. The team cordons and drains the node and pauses automatic sync, then locks the pause so that no tool restarts the node's work. The procedure ends with a release step that removes the lock after the disk is replaced.
Violating the rule. The team starts the repair with automatic sync still on. Halfway through, sync overwrites the manual fix and restarts a service on the node. The work is destroyed and the engineer is surprised by the restart.
Management variant.
Scenario. A product team commits to a sprint goal for two weeks.
Applying the rule. The team protects the sprint goal from unplanned changes. New requests wait for the next sprint. Only the Product Owner, who is accountable for the goal, may cancel the sprint.
Violating the rule. Stakeholders add urgent tasks to the sprint every few days, and anyone with influence can redirect the team. The committed goal is not protected, and nobody accountable decided to cancel it.
Real case: DuPont Yerkes tank explosion (2010) (2010).
At the DuPont Yerkes plant in Buffalo, New York, a contract welder repaired an agitator support on top of a storage tank. Flammable vapor flowed into the tank from connected tanks that were still in service, and it ignited. The welder died. The overflow line between the tanks was never blinded, and DuPont did not isolate the tank before it authorized the hot work.
What it shows for P66. The lockout card for the tank covered its five valves and the agitator motor, but not the overflow line, which had no valves and was never blinded. That line still joined the tank to the tanks in service. Isolation has to cover every input, including vent and overflow lines, and the energy control procedure has to list each one.
The welder died instantly from blunt force trauma
when a contract welder and foreman were repairing the agitator support atop an atmospheric storage tank containing flammable vinyl fluoride.
flammable vinyl fluoride (VF) vapor from interconnected, in-service process tanks flowed undetected into the tank and ignited when the welder was repairing the agitator support assembly.
The overflow line to all three tanks was never blinded, which provided a direct flow path from the vapor space in tanks 2 and 3 to the out-of-service tank 1.
DuPont did not properly isolate and lockout tank 1 from in-service tanks 2 and 3 prior to authorizing hot work on tank 1.
DuPont was fined for not including steps to isolate the slurry tank overflow line in their energy control procedure, and for not installing blinds in the overflow line.
all five valves leading to and from tank 1 and the agitator motor had been locked out.
The card instructed workers to try to start the motor before beginning work, but did not reference the overflow line, which had no valves.
P67. Critical path¶
The rule: identify the longest chain of dependent tasks (critical path) and protect it first
Illustrative, not evidence.
Scenario. A team migrates a service to a new cluster. The tasks are data sync, DNS cutover, load testing, a new dashboard and documentation.
Applying the rule. The plan marks the longest chain of dependent tasks, data sync then DNS cutover, as the critical path. The plan also states the slack of load testing, the dashboard and the documentation. When people are short, the team protects the critical path first.
Violating the rule. The team adds people to the dashboard and the documentation, which have slack, while data sync slips. The migration date moves because the critical path is delayed.
Management variant.
Scenario. A company moves its office to a new building. The tasks are network installation, furniture delivery, badge setup and signage.
Applying the rule. The plan marks network installation and the tasks that depend on it as the critical path. Furniture and signage are marked with their slack. The project lead protects the critical path first.
Violating the rule. The lead pushes the signage supplier to finish early, which gains nothing, while the network installation slips and delays the move.
P68. Observational method¶
The rule: design for the most probable conditions, monitor the real values, and plan trigger levels and backup actions in advance
Illustrative, not evidence.
Scenario. A team rolls out a new version of a service, and the real error rate under production load is unknown until the rollout runs.
Applying the rule. The team designs for the most probable load and monitors the real error rate. It sets a trigger level of 2 % errors and plans a backup action in advance, which is an automatic rollback. When the error rate crosses 2 %, the rollback runs.
Violating the rule. The team monitors the error rate but sets no trigger level. At go-live, the error rate climbs, and the team finds that nobody has planned a rollback.
Management variant.
Scenario. A programme migrates data to a new platform, and the effort needed stays uncertain until the work is under way.
Applying the rule. The programme sets a time tolerance and a planned response before the work starts. It monitors progress. When the migration will exceed its time tolerance by 10 %, the steering group decides on scope.
Violating the rule. The programme sets no tolerance and plans no response. When the migration runs late, nobody knows who decides or what the options are.
Real case: Nicoll Highway collapse, Singapore (2004) (2004).
On 20 April 2004 the temporary retaining wall of a deep excavation for a metro line collapsed next to the Nicoll Highway in Singapore. Four people died. The interim report of the Committee of Inquiry found that the instrument monitoring was not effective, and that critical strut load readings were wrong.
What it shows for P68. The interim report called for instrument monitoring that is effective, adequately resourced and maintained, and for senior managers able to suspend or stop the work when readings exceed the pre-determined risk levels. Trigger levels only help when the readings are right and someone acts on them.
At about 3.30pm on 20 April 2004, the temporary retaining wall system for a deep excavation adjacent to the Nicoll Highway collapsed.
to construct a cut and cover tunnel for the Mass Rapid Transit (MRT) Circle Line project C824
Among them, four persons died (including an LTA technical officer), and several others were injured in the accident.
Instrument-based performance monitoring systems must be effective, adequately resourced and maintained unlike in C824.
The strut load readings provided by one of the ‘specialist’ personnel were critical to the safety monitoring of the retaining wall system. These were relied on extensively by the main contractor and LTA, but were subsequently found to be erroneous at the COI.
These senior managers must be experienced enough to make the right judgement call either to suspend or to stop the work when instrumentation data exceed the pre-determined risk levels.
P69. Guided change¶
The rule: make the need visible, form a group that leads the change, communicate the vision far more often than seems necessary, remove obstacles, plan early visible wins, and continue until the new way is part of daily routine
Illustrative, not evidence.
Scenario. An engineering organization moves all its teams to a new CI system.
Applying the rule. The organization writes down the reason for the change and names a leading group. It sends weekly updates, defines early wins from a pilot team and switches the old system off only when all teams use the new one. The end criterion, the old way is switched off, is written down.
Violating the rule. A manager announces the new CI system by email and declares the change done. There is no leading group, no communication plan and no end criterion, so teams keep using the old system.