Stale runbooks are an incident risk: the wrong doc at 3am costs more than every review you skipped

Price the risk with three numbers: your cost per outage minute, how long a wrong runbook step burns, and how often runbooks are consulted. A single wrong step mid-incident typically costs twenty to forty minutes: execute it, watch it fail or make things stranger, unwind it, then improvise the correct action under pressure, often waking the one person who knows. If an outage minute costs your business hundreds of dollars, which is modest for anything revenue-facing, one wrong step is a four to five figure event.

Against that, the prevention is a quarterly verification pass at fifteen to thirty minutes per runbook. The asymmetry is stark: a year of verifying one runbook costs about two engineer-hours, less than a single wrong step burns in the middle of a Sev1. Staleness reviews look optional only because the invoice arrives during an incident, itemized under MTTR where nobody attributes it to the doc. The broader accounting, including the questions-and-onboarding terms, is in the cost of outdated documentation.

The clearest public illustration is GitLab's database outage of January 31, 2017. Their published postmortem records that when the team reached for recovery, none of their five backup and replication mechanisms was working reliably, a fact discovered during the incident rather than before it. Documented procedures existed; what did not exist was recent evidence that they worked. Recovery ran on a copy of the database that happened to exist for other reasons, and hours of production data were lost.

The general lesson from postmortem culture is the same one the SRE literature draws about backups: a procedure that has not been exercised recently is not a procedure, it is a hypothesis. Written-but-unverified is the exact state a stale runbook occupies, and incident reviews across the industry keep finding it because nothing in normal operation ever tests the document. The failure surfaces only when the document is load-bearing.

The counter-measure is to make verification visible and scheduled. Stamp every runbook with a last_verified date and the name of the person who verified it, in front matter where a CI check can read it, and hold the hard line on what verified means: someone executed every step against staging and each behaved as written. A read-through does not reset the stamp, because plausibility is exactly what stale runbooks are good at.

Drills are how the stamps stay honest. A quarterly game day where on-call engineers execute runbooks literally, ideally engineers who did not write them, converts 3am discoveries into 10am findings. A drill that fails is a success: the runbook was wrong, and you found out on a calendar instead of a pager. Wire the stamp ages into a CI freshness gate so a runbook sliding past its window fails a build instead of failing an incident.

---
service: payments-api
scope: Sev1-Sev2 database failover
last_verified: 2026-07-03
verified_by: dana
next_drill: 2026-10-01
---
# Payments API: primary database failover
#
# "Verified" means every step below was executed against staging
# and behaved as written. A read-through does not reset the date.

Every incident is a free, brutal verification pass, and most postmortem templates waste it. Add two questions: did the runbook match reality, and which step was missing, wrong, or ambiguous. Any gap becomes an action item with an owner and a date, exactly like a code fix, and the runbook edit ships while the memory is fresh. Teams that run this loop get runbooks verified by their most realistic possible test, an actual incident, at zero additional cost.

The same review should ask whether the supporting material held up, because a runbook is more than its numbered steps: the diagram at the top, the dashboard links, the escalation names all rot on their own schedules. An escalation pointing at someone who left, or a dashboard link that 404s, burns the same minutes a wrong step does.

The topology diagram is the runbook component most likely to be silently wrong, because it changes when infrastructure changes, not when procedures do, and nobody re-exports a wiki image after a Terraform merge. An on-call engineer navigating a failover with a diagram showing last year's replica layout is being misled by the most authoritative-looking thing on the page.

The fix is to stop pasting exports: embed the diagram by URL so the runbook shows the current version, within minutes of an edit, and put regeneration one command away with an agent connected to the Datadef MCP server that redraws it from what the repository declares. Two honest limits: live embeds require the project to be shared public, so runbooks for confidential systems should instead pair an exported image with its own last_verified stamp, and nothing watches the repo for you: regenerating after infra merges is a CI job or a review habit, which is exactly the kind of trigger the drill calendar already gives you.

• Keep incident runbooks up to date
• Keep runbooks in sync with infrastructure
• The cost of outdated documentation
• Documentation SLAs