Operations

What this capability covers

Operations is the engineering that begins where a launch ends: keeping a system understandable while it runs, safe to change, and recoverable when something fails.

This capability covers site reliability engineering, observability, DevOps practice, production operations, incident readiness, change safety, operational automation, and recovery and operability engineering.

Reliability is treated as a property of the system’s design rather than a service wrapped around it afterwards.

Typical engineering problems

• Monitoring that reports component health while users are experiencing an outage.
• Dashboards that show a problem exists but not where it is.
• Deployments treated as events to be scheduled rather than routine operations.
• Incident response that depends on who happens to be awake.
• Recovery procedures that are documented but have never been executed.

Engineering approach

• Instrument for questions, not for graphs. Telemetry earns its place by answering something during an incident.
• Make change routine. Small, reversible, frequent changes are safer than large, careful, rare ones.
• Design for recovery. How a system is restored is a design requirement, and a procedure that has never been rehearsed is an assumption.
• Automate the repeated response. Anything done the same way twice under pressure is a candidate for automation.
• Treat incidents as information. The output of an incident is a change to the system, not only a report about it.