H. Operation and maintenance¶
How to run, watch and maintain the system.
Take the Operation and maintenance quiz when you have read the examples.
P70. Monitoring in operation¶
The rule: measure its behavior in real use, and alert on symptoms that users feel; where the output stays perfect until a sudden collapse (cliff effect), also monitor the remaining margin
Illustrative, not evidence.
Scenario. A team runs an online checkout service. Its dashboards show CPU, memory and disk for each host, but nothing about what shoppers experience.
Applying the rule. The team adds a dashboard and alerts for the checkout failure rate and page latency, the symptoms shoppers feel. Because the payment queue runs fine until it suddenly fills, it also tracks the queue's remaining headroom. Each service has user-facing dashboards and alerts.
Violating the rule. Shoppers cannot pay for two hours. The team learns about it from a flood of support emails, because every host metric looked normal.
Management variant.
Scenario. A department head reads a weekly report of last month's results and nothing else.
Applying the rule. The head tracks leading indicators as well as the lagging results. Each quarter the head also joins an on-call shift, to see where the work happens.
Violating the rule. The head reads only the report. A growing backlog of complaints stays invisible until the quarterly results drop.
Real case: Cloudflare global outage from a firewall rule (2019) (2019).
On 2 July 2019 Cloudflare deployed a new rule to its web application firewall. The rule exhausted the CPUs that serve HTTP and HTTPS traffic across its whole network, and visitors saw 502 errors. The procedure for rule changes allowed a global push without a staged rollout, so the change went global in seconds. Synthetic tests paged the team three minutes later, but the alert for the global traffic drop was slow.
What it shows for P70. Tests that act like users, run from outside the network, caught the fault within minutes. The alert on the drop in user traffic took too long, and the postmortem lists it as a weakness.
On July 2, we deployed a new rule in our WAF Managed Rules that caused CPUs to become exhausted on every CPU core that handles HTTP/HTTPS traffic on the Cloudflare network worldwide.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
This resulted in our customers (and their customers) seeing a 502 error page when visiting any Cloudflare domain.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
The SOP allowed a non-emergency rule change to go globally into production without a staged rollout.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
However, in this case, that speed meant that a change to the rules went global in seconds.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
Changing the SOP to do staged rollouts of rules in the same manner used for other software at Cloudflare while retaining the ability to do emergency global deployment for active attacks.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
Three minutes later the first PagerDuty page went out indicating a fault with the WAF. This was a synthetic test that checks the functionality of the WAF (we have hundreds of such tests) from outside Cloudflare to ensure that it is working correctly.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
The first alert for the global traffic drop took too long to fire.
Source: Cloudflare: Details of the Cloudflare outage on July 2, 2019
P71. Checklists¶
The rule: use a short checklist, and confirm each item by pointing at it and saying its status aloud
Illustrative, not evidence.
Scenario. Once a year, an operations team upgrades the major version of its main database. The work is complex, rare and done in a short maintenance window.
Applying the rule. The team uses a one-page checklist. Each item is confirmed by pointing at it and saying its status aloud. After the last upgrade's postmortem, the team added the backup-verification step that was missed.
Violating the rule. An engineer does the upgrade from memory. A step is skipped, and nobody notices until the service fails to start.
Management variant.
Scenario. An office hires and loses staff a few times a year, so offboarding is a rare procedure.
Applying the rule. An offboarding checklist lists access removal, knowledge handover and equipment return. Each item is confirmed aloud, and the list is updated after any slip.
Violating the rule. A manager offboards a leaver from memory. The leaver's access to the finance system stays active for months.
Real case: Gulfstream G-IV runway overrun at Bedford (2014) (2014).
A Gulfstream G-IV tried to take off from Bedford, Massachusetts, with its gust lock still on, so the flight controls were locked. The crew had not done the flight control check before takeoff. Seven people died. Recorded data showed the pilots had skipped complete flight control checks before 98% of their previous 175 takeoffs.
What it shows for P71. The checklist existed, but the crew did not use it, and the skipped check had become a habit. A checklist only protects when every item is done and confirmed, every time.
The two pilots, a flight attendant, and four passengers died.
their attempt to take off with the gust lock system engaged
During the engine start process, the flight crew neglected to disengage the airplane’s gust lock system
before initiating takeoff, the pilots neglected to perform a flight control check that would have alerted them of the locked flight controls.
the pilots had neglected to perform complete flight control checks before 98% of their previous 175 takeoffs in the airplane, indicating that this oversight was habitual and not an anomaly.
Contributing to the accident were the flight crew’s habitual noncompliance with checklists
P72. Critical control points¶
The rule: define those critical control points, rank them by consequence rather than by how often incidents happen, and give each one a limit, monitoring, a predefined corrective action, a record and regular verification in practice
Illustrative, not evidence.
Scenario. A bakery pasteurizes cream filling on every batch. The few steps that decide safety are the heating step and the cooling step.
Applying the rule. The bakery names the heating and cooling steps as critical control points and ranks them by the harm a failure could cause. Each has a limit, a probe that monitors it, a predefined corrective action, a log, and a verification of the probe done this quarter.
Violating the rule. The bakery logs all forty process readings the same way, with no defined limits. Nobody knows which reading matters or what to do when it moves.
Management variant.
Scenario. A finance team closes the books each month.
Applying the rule. The team identifies the few controls that prevent a material misstatement, such as the bank reconciliation. Each gets a limit, an owner, monitoring and a regular test.
Violating the rule. The team reviews every ledger line with the same effort. The control that matters most has no owner and was never tested.
P73. Response faster than harm¶
The rule: make sure the time to detect + alert + respond + recover is shorter than the time until harm, with a margin
Illustrative, not evidence.
Scenario. A log service fills its disk at a steady rate. The database stops when the disk is full, three hours after the leak starts.
Applying the rule. The team measures detection at 10 minutes, response at 20 minutes and automated recovery at 30 minutes. That is one hour in total, well inside the three hours, so the margin is large.
Violating the rule. The alert fires only when the disk is already full. Detection, response and recovery all start after the harm has begun.
Management variant.
Scenario. A company can tolerate a payroll outage of at most 5 days.
Applying the rule. The company sets the recovery time objective to 3 days, inside the 5-day maximum tolerable disruption, and tests that recovery finishes within the 3 days.
Violating the rule. Nobody tests payroll recovery. When a failure happens, restoring takes 8 days and staff go unpaid.
Real case: Knight Capital trading loss (2012).
Knight Capital deployed new trading code to its order router, SMARS. The code reused a flag once used by an old function, Power Peg. Knight had stopped using Power Peg years earlier, but its code was still present and callable. A technician did not copy the new code to one of the eight servers, and no second technician reviewed the deployment. On 1 August 2012, in about 45 minutes, the old code on that server sent millions of orders into the market, while staff tried to find the source of the problem. Before the market opened, an internal system had sent 97 e-mails that pointed at the fault, but they were not designed as alerts. Knight lost 460 million dollars.
What it shows for P73. Knight's system kept sending orders while its staff tried to find the source of the problem, and the positions built up over 45 minutes. The SEC found that Knight needed clear guidance on when to disconnect a malfunctioning system from the market. A response decided in advance, such as when to disconnect, is what lets the response beat the harm.
On August 1, 2012, Knight Capital Americas LLC (“Knight”) experienced a significant error in the operation of its automated routing system for equity orders, known as SMARS.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
This unused code previously had been used for functionality called “Power Peg,” which Knight had discontinued using many years earlier.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Despite the lack of use, the Power Peg functionality remained present and callable at the time of the RLP deployment.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
The new RLP code also repurposed a flag that was formerly used to activate the Power Peg code.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
resulting in 4 million executions in 154 stocks for more than 397 million shares in approximately 45 minutes.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Ultimately, Knight realized a $460 million loss on these positions.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
SMARS routed millions of orders into the market over a 45-minute period
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server, nor the new RLP code added.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system sent 97 of these e-mail messages to a group of Knight personnel before the 9:30 a.m. market open.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight did not design these types of messages to be system alerts, and Knight personnel generally did not review them when they were received.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight’s system continued to send millions of child orders while its personnel attempted to identify the source of the problem.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
Knight needed clear guidance for its technology personnel as to when to disconnect a malfunctioning system from the market.
Source: SEC: In the Matter of Knight Capital Americas LLC, Release No. 34-70694 (2013)
P74. Maintenance by failure pattern¶
The rule: choose the maintenance method by how each part actually fails: replace on a schedule only what wears out with age, and monitor the condition of everything else
Illustrative, not evidence.
Scenario. A plant keeps a fleet of pumps. Bearings wear with age. The control electronics fail at random.
Applying the rule. The plan replaces the bearings on a schedule and monitors the vibration and temperature of the rest. Each task states the failure pattern it addresses: wear-out or random failure.
Violating the rule. The plan replaces every pump part every six months, the electronics included, just in case. No task says which failure pattern it addresses, and the electronics still fail at random between replacements.
Management variant.
Scenario. An organization reviews access rights for its systems.
Applying the rule. Access to high-risk systems is reviewed every quarter and access to low-risk tools once a year, according to how each control actually fails.
Violating the rule. Every control gets one fixed annual review. The high-risk control drifts for months between reviews.
Real case: Alaska Airlines flight 261 (2000) (2000).
Alaska Airlines flight 261 lost pitch control and crashed into the Pacific Ocean. Everyone on board died. The acme nut threads in the stabilizer trim jackscrew failed from excessive wear. The lubrication and end play check intervals had been extended without a task-by-task analysis.
What it shows for P74. The acme nut threads wear, and the end play check measures that wear before it becomes a failure. The lubrication and end play check intervals were extended without a task-by-task analysis, which let the threads wear to failure without the opportunity for detection. The NTSB warned that an absence of maintenance history is not enough to justify a longer interval. Each maintenance task needs to rest on how its part actually fails.
The 2 pilots, 3 cabin crewmembers, and 83 passengers on board were killed
crashed into the Pacific Ocean about 2.7 miles north of Anacapa Island, California
the probable cause of this accident was a loss of airplane pitch control resulting from the in-flight failure of the horizontal stabilizer trim system jackscrew assembly
The thread failure was caused by excessive wear
nut threads to wear to failure without the opportunity for detection
In general, the absence of maintenance history should not be considered adequate justification to extend the interval for the performance of a critical maintenance task.
the escalation of C-check intervals in the MSG-3 MD-80 MRB did not involve a task-by-task analysis of each task
resulted in significant extensions of both the lubrication and end play check intervals without any such analysis or support.
the existing process by which manufacturers revise recommended maintenance task intervals and by which airlines establish and revise these intervals does not include task-by-task engineering analysis and justification
P75. Actionable alerts¶
The rule: make every alert actionable, and remove or change alerts that nobody acts on
Illustrative, not evidence.
Scenario. An on-call team receives alerts from its monitoring system in one channel.
Applying the rule. Every alert has an owner and a runbook action. The team measures the share of alerts that led to no action, keeps it low, and deletes or changes the noisy alerts.
Violating the rule. The alert channel gets hundreds of messages a day. Nobody reads it, so a real alert is lost among the noise.
Management variant.
Scenario. A programme board receives a status report each week.
Applying the rule. The weekly status shows only items outside tolerance. Team leads escalate when a tolerance is forecast to be breached, and reports that lead to no action are dropped.
Violating the rule. The board gets a forty-page report every week. It leads to no decisions, and the one overdue item is on page 31.
Real case: Knight Capital trading loss (2012). Read the full case above.
What it shows for P75. Before the market opened, the system sent 97 e-mails that pointed at the fault. They were not designed as alerts, and staff generally did not review them. A message that nobody acts on does not protect anyone.
P76. Sterile critical phase¶
The rule: allow only the activity and communication that the critical phase needs
Illustrative, not evidence.
Scenario. A team runs a production cutover with a bridge call of fifteen people.
Applying the rule. The team has defined cutover as a critical phase and written down its communication rules. Only the cutover steps and their status updates are allowed on the bridge.
Violating the rule. During the cutover, people hold side discussions on the bridge, and one of them starts a parallel deployment of an unrelated change.
Management variant.
Scenario. A product team has meetings spread over every day, and its engineers are interrupted often.
Applying the rule. The team protects blocks of uninterrupted work: no meetings on Tuesday and Thursday mornings. In a crisis it uses one channel with one commander.
Violating the rule. Messages and meetings can reach engineers at any time. During a crisis, each department opens its own channel.
Real case: Colgan Air flight 3407 (2009) (2009).
Colgan Air flight 3407 stalled on approach to Buffalo, New York, and crashed into a house. Everyone on board and one person on the ground died. The NTSB found that the captain responded inappropriately to the stick shaker. One contributing factor was that the crew did not follow sterile cockpit procedures. They talked about unrelated things during the final minutes of the descent.
What it shows for P76. The sterile cockpit rule keeps attention on the task during critical phases. Here the pilots chatted through the descent, and the NTSB found the talk distracted them from their operational tasks.
The 2 pilots, 2 flight attendants, and 45 passengers aboard the airplane were killed, one person on the ground was killed
was on an instrument approach to Buffalo-Niagara International Airport, Buffalo, New York, when it crashed into a residence in Clarence Center, New York
the probable cause of this accident was the captain’s inappropriate response to the activation of the stick shaker, which led to an aerodynamic stall from which the airplane did not recover.
the flight crew’s failure to adhere to sterile cockpit procedures
The pilots were involved in nonpertinent conversation during all phases of flight, including those that are defined as critical by the sterile cockpit rule.
the pilots’ nonpertinent conversation during the final minutes of the descent (which included a discussion of their previous experiences flying in icing conditions) distracted them from their operational tasks.
The primary reason for 14 CFR 121.542, otherwise known as the sterile cockpit rule, is to ensure that a pilot’s attention is directed to operational concerns during critical phases of flight and is not redirected or degraded because of nonessential activities or conversation.
P77. Data as a dependency¶
The rule: version and validate the data like code, monitor for drift, and retrain or re-evaluate when the world changes
Illustrative, not evidence.
Scenario. A bank team runs a model that scores payments for fraud. Customer behavior changes over the year.
Applying the rule. The data and the model have versions, and incoming data is validated. A drift monitor alerts when the score distribution moves, and the team re-evaluates the model when it fires.
Violating the rule. The model is trained once and then left alone. Over several months it silently gets worse, and nobody notices.
Management variant.
Scenario. The board report and the HR system both give a headcount figure.
Applying the rule. The headcount definition is versioned and shared. Data quality is checked, and the figure is watched for drift, so both reports show the same number.
Violating the rule. Each team defines headcount in its own spreadsheet. The board sees a figure that differs from HR's, and nobody knows why.
P78. Asset inventory¶
The rule: keep a complete inventory of all assets, with owner, condition and lifecycle stage
Illustrative, not evidence.
Scenario. A company runs hundreds of cloud servers and test environments across many teams.
Applying the rule. Each asset has an owner, a condition and a lifecycle stage in the inventory. An automated discovery compares the real assets with the inventory, and it finds no unknown asset.
Violating the rule. A test cluster is forgotten after a project ends. It has public access, is not in any inventory, and nobody patches it.
Management variant.
Scenario. A company holds many supplier contracts.
Applying the rule. A contract register lists the owner, status and lifecycle stage of every contract, including end dates and notice periods. It is compared with the payment records to find contracts it missed.
Violating the rule. Each department keeps its own list. A contract renews automatically because nobody had its notice period.
Real case: Equifax data breach (2017) (2017).
In 2017 attackers stole the personal data of about 148 million people from Equifax. They entered through an Apache Struts vulnerability in a legacy application. Equifax did not have a complete, up-to-date inventory of its IT assets or the software on them. The executive responsible for the application did not know it ran Apache Struts.
What it shows for P78. If an organization does not know what is on its networks, the report says, it will not know where patching is needed. Equifax had inventories, but none was comprehensive, and the executive responsible for the affected environment did not know it ran Apache Struts. P78 asks for a complete inventory with an owner for each asset.
On September 7, 2017, Equifax announced a cybersecurity incident affecting 143 million consumers.
This number eventually grew to 148 million
Attackers enter the Equifax network through the Apache Struts vulnerability located
The company’s lack of knowledge about the software used within its legacy IT environment was a key factor leading to the 2017 data breach.
As of July 2017, the company did not have a comprehensive and up-to-date inventory of its IT assets or the software operating on its systems.
If an organization does not know what is on its networks, it will not know where patching is needed.
at the time that the breach was announced, I wasn’t even aware that we were running Apache Struts in the particular environment.
inventories existed, but they weren’t comprehensive.
Payne, who had ultimate responsibility for the ACIS environment, stated
P79. Span of control¶
The rule: keep the number of people one person directs in a manageable range (guideline: about five), and add group leads above that
Illustrative, not evidence.
Scenario. A major incident needs many people to work together in one channel.
Applying the rule. The incident commander directs five people. Above that, group leads each direct a small team. The incident record lists the roles and how many people each lead directed.
Violating the rule. One incident commander coordinates 20 people directly in one channel. Orders overlap, and tasks are missed.