Docs & Code Guide

Keep incident runbooks up to date: make the post-incident review the mandatory update trigger

An incident runbook is read under the worst possible conditions: at 3am, by whoever is on call, while something is down. A stale step is not an inconvenience there, it is added minutes of outage. The good news is that incident response already contains its own update ceremony. Every incident ends in a review, and the review is the one moment when everyone knows exactly which runbook steps were wrong. The whole discipline is refusing to close that review until the runbook edits are merged.

Try it free
7 min readFor SRE and on-call teams whose runbooks failed them mid-incident at least once

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

200/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

The update trigger you already run

Most documentation lacks a natural update moment; incident runbooks have one built in. The post-incident review reconstructs what happened minute by minute, which includes every point where the runbook lied: the dashboard link that 404ed, the rollback step that referenced a deleted job, the threshold that no longer matches the alert. The established practice, the one PagerDuty's postmortem process and incident.io's follow-up tracking both encode, is capturing those as review action items with owners and due dates, tracked like any other work.

The discipline that separates teams whose runbooks improve from teams whose runbooks rot is one rule: a review with runbook lies in the timeline does not close until the fixes are merged. Filed-but-forgotten follow-ups are how the same runbook fails twice, and the second failure is the one people remember when they stop reading runbooks entirely, the spiral described in stale runbooks as incident risk.

Last-verified stamps, refreshed by drills

Incidents only exercise the runbooks that fire. The payments runbook used monthly stays sharp; the region-failover runbook used never decays in silence, and it is the one whose failure would matter most. The mechanism for the quiet ones is a last-verified stamp plus a drill calendar: a game day where someone executes the runbook against reality, or a controlled environment, and updates the stamp. The stamp records verification, not editing; a runbook edited yesterday but never executed since the infrastructure migrated is unverified where it counts.

The stamp also gives you a queryable freshness inventory: sort runbooks by last-verified date and the top of the list is your drill backlog. Teams that formalize the acceptable age per tier are implementing a documentation SLA, with the pager as the enforcement mechanism.

# Runbook: API 5xx spike

**Last verified:** 2026-08-03 (game day, executed end to end)
**Owner:** platform on-call
**Trigger:** alert `api-5xx-rate` above 2% for 5 minutes

1. Check the deploy feed. If a deploy landed in the last 30 minutes,
   roll back first and diagnose second.
2. Open the API golden-signals dashboard (link tested in drills).
3. If error rate persists after rollback, page secondary and open
   an incident channel.

The on-call handoff as a freshness checkpoint

Between incidents and drills there is a third rhythm already on the calendar: the on-call handoff. The outgoing engineer is the person with the freshest knowledge of which runbooks were touched, which were consulted and found wanting, and which alerts fired with no runbook at all. Five minutes of the handoff spent recording that, three checkbox questions in the handoff note, converts a week of on-call experience into a maintenance signal instead of losing it.

The handoff catches a class of decay the review misses: near-miss staleness, where the on-call engineer noticed a wrong step but worked around it without an incident. Those workarounds are exactly the edits that prevent the next incident from being longer.

Incident runbooks are not ops runbooks

This page is about incident response paths specifically: the documents on the other end of a page, executed under time pressure. General operational runbooks, the routine procedures for deploys, rotations and maintenance, share the format but not the failure mode, and their sync problem is tied to infrastructure change rather than incident learning; that loop is covered in keep runbooks in sync with infrastructure.

The distinction matters for the quality bar. An ops runbook can afford a slightly stale screenshot; an incident runbook cannot afford an unverified step, because verification during execution costs outage minutes. That is why the incident set gets the stricter loop: mandatory review-driven edits, drills for the rarely-fired paths, and a freshness policy with dates, the shape described in docs freshness policy.

The escalation picture next to the steps

Runbooks are prose plus one picture: the escalation and system context diagram that tells the 3am responder what is upstream, what is downstream, and who gets paged next. That diagram decays like any other, and it has the same fix: keep it in Datadef, regenerate it when the paging structure or the architecture changes, by hand or through an agent connected to the Datadef MCP server, registry name io.datadef/mcp. Datadef does not watch anything by itself; the update is one agent call, and the agent signs in with a paid-plan API key.

A project shared public serves the diagram at a permanent image URL that follows edits within minutes, which suits a runbook index in a public-friendly wiki. Internal-only incident tooling is the more common case, and there the pattern is a private project plus a dated export in the runbook, refreshed at drill time along with the stamp. The prose steps stay yours to maintain: Datadef keeps the picture current, not the procedure.

FAQ

How do I keep incident runbooks up to date?

Attach the updates to the ceremonies you already run. Every post-incident review captures runbook errors as action items with owners, and the review does not close until they merge. Rarely-fired runbooks get scheduled drills that execute them and refresh a last-verified stamp. The on-call handoff records which runbooks were consulted and found wanting during the week.

Who should update a runbook after an incident?

Someone who was in the incident, assigned during the review while the memory is fresh, usually the responder who hit the wrong step. The review facilitator's job is making sure each runbook error in the timeline becomes an owned action item, and the team's job is treating those items as blocking the review's closure rather than as optional cleanup.

What is a last-verified stamp on a runbook?

A dated line recording the last time someone executed the runbook against reality, in an incident or a drill, as opposed to the last time someone edited the text. The distinction matters because an edited-but-never-executed runbook can still be wrong everywhere it counts. Sorting runbooks by the stamp produces the drill backlog.

How often should incident runbooks be tested?

Frequently-fired runbooks are tested by reality and need no schedule. For the rest, set a drill cadence by tier: quarterly for critical paths like failover and data recovery, twice a year for the long tail. The drill is an engineer executing the runbook end to end and updating the stamp, and it routinely finds broken links, renamed jobs and drifted thresholds.

Are incident runbooks different from operational runbooks?

Same format, different failure mode. Incident runbooks are executed under pager pressure, where an unverified step costs outage minutes, so they get the stricter loop: review-driven updates, drills, verification stamps. Operational runbooks for routine procedures decay with infrastructure change instead, and their sync loop ties to infrastructure merges rather than incident reviews.