Site Reliability Engineering
Also called: SRE, reliability engineering
Treating reliability as an engineering problem with measurable targets and a budget for failure, rather than as an aspiration to never break.
The useful idea in SRE is that perfect reliability is the wrong goal. Chasing it is enormously expensive and, past a point, users cannot tell the difference.
So you pick a target — a service level objective — and treat the gap between that target and 100% as a budget you are allowed to spend.
Error budgets, concretely
If your SLO is 99.9% monthly availability, you have roughly 43 minutes of allowed downtime a month. That is the budget.
Spend it slowly and you can keep shipping features. Burn through it in the first week and the next work item is stability, not features. It converts an argument about priorities into arithmetic, which is why it works.
Why the target should be user-facing
"CPU below 80%" is not an SLO. "Checkout completes successfully for 99.9% of attempts" is. The second one is something a user would notice and something you can be paged about at 3am and act on.
This is also the difference between alerting that works and alerting that gets muted.
The practices that matter more than the label
- Alerts tied to user-visible symptoms, each with a runbook written before it fires
- Blameless post-mortems that produce a landed fix, not a document
- Toil measured and automated away rather than absorbed
- Recovery paths rehearsed — a restore that has never been executed does not work
Do you need an SRE function?
Below roughly fifty engineers, usually not as a separate team — you need the practices, owned by whoever runs the platform. Hiring two SREs also does not give you 24x7 coverage, which is the arithmetic that leads many teams to NoOps instead.
Related terms