Keep runbooks in sync with infrastructure: link steps to living sources

The failure modes are concrete. A hostname copied into step 3 stops resolving after a migration to new instances. A link to "the latency dashboard" 404s because someone reorganized Grafana folders and the URL embedded the title, not the UID. A step says to assume the ops-admin role, but a security review narrowed that role and the command now fails with an error the runbook never mentions. Each of these worked when written; the infrastructure moved and the copies did not.

The consequences compound at the exact wrong time. The engineer following the runbook is paged, unsure, possibly new to the service, and every dead end costs minutes of incident time and a piece of their trust in every other runbook. The organizational math of that risk is laid out in stale runbooks as incident risk.

Anything Terraform created, Terraform can name. A runbook step that says "connect to the bastion" should not contain a hostname; it should contain the command that resolves the hostname from the state that created it. The value the on-call engineer copies is then correct by construction, because it comes from the same source the infrastructure does.

The same principle covers dashboards. Grafana URLs contain a stable UID and a cosmetic slug: /d/a8x2kq3/payments-queues survives any retitle, while a link built from the title dies at the first rename. Link the UID form. For IAM, name the role from the IaC definition and link the policy file, so the runbook and the permission change in the same review when security tightens things.

---
owner: payments-oncall
last_verified: 2026-08-02   # game day GD-14
---

## 3. Connect to the bastion

Do not use a remembered hostname. Resolve it from the state
that created it:

    terraform -chdir=infra/prod output -raw bastion_hostname

## 4. Check queue depth

Dashboard (UID permalink, survives folder moves and renames):
https://grafana.internal/d/a8x2kq3/payments-queues

A runbook is only verified by execution. A game day, a scheduled exercise where an engineer follows the runbook against a real or staged failure, is to runbooks what CI is to code: the test that finds the renamed host before 3am does. Run one per critical runbook per quarter, have someone unfamiliar with the service drive, and treat every point where they stop and ask a human as a defect to fix in the document before the exercise ends.

Stamp the result. A last_verified date with a reference to the exercise, as in the snippet above, tells the next reader exactly how much to trust each step. A runbook verified last month reads differently from one verified never, and making that visible is most of the battle.

Runbooks decay per service, so ownership must be per runbook, and the natural owner is the rotation that gets paged for it. The working agreement: whoever is on call when a runbook step fails fixes that step before the retro closes, and each runbook is re-verified on a stated cadence. What that cadence should be, and what to do when it slips, is the subject of a documentation SLA.

Most runbooks open with an architecture sketch, and it is usually the stalest part of the page: a screenshot from the wiki, itself a screenshot of something older. Replace it with a live embed. A Datadef diagram shared public serves a permanent image URL that Confluence, Notion, or Markdown render like any image, and an edit to the diagram shows up in the runbook within minutes. When infrastructure changes merge, an AI agent connected to the Datadef MCP server can update the diagram from the repo as a CI step; nothing watches the repo by itself, the regeneration is one wired command.

The honest limits: the embed requires a public project, which may not fit infrastructure diagrams that are themselves sensitive, and Datadef updates the diagram, not the runbook prose. The steps, the commands, and the last_verified stamp remain human work with the mechanisms above.

• Keep incident runbooks up to date
• Stale runbooks are incident risk
• Keep infrastructure docs in sync with Terraform
• A documentation SLA