Documentation drift metrics: measure divergence from reality, not age, with scriptable checks

The boundary with freshness matters because the two metrics answer different questions and trigger different actions. Freshness, covered in documentation freshness metrics, asks when was this touched and is computed from timestamps alone; it is cheap, continuous, and blind to content. Drift asks is this still true and requires comparing the document against the system it describes; it is more expensive per check and it is the metric readers actually feel. A freshness dashboard flags suspects; drift checks convict.

The concept itself, why divergence accumulates and what it does to a team, belongs to the anchor guide on documentation drift. This page is the measurement layer: signals you can script and put on a dashboard.

The strongest cheap signal is a dangling identifier: a service, table, topic, or config key that the docs mention and the codebase no longer contains. Renames and removals leave these behind reliably, and each one is a concrete lie a reader can hit. The check is grep-shaped: harvest code-formatted terms from the docs, then ask git grep whether each still exists in the source tree.

Expect noise on the first run, prose words caught in backticks, vendored names, and tune with an allowlist. The count matters less than the trend: dangling identifiers per 100 pages, week over week, is a drift rate.

# Dangling identifiers: mentioned in docs, absent from the codebase.
# Harvest backticked terms from docs, check each against git grep.
grep -rhoE '`[A-Za-z][A-Za-z0-9_.-]{2,}`' docs/ \
  | tr -d '`' | sort -u \
  | while read -r name; do
      git grep -q --fixed-strings "$name" -- src/ services/ infra/ \
        || echo "dangling: $name"
    done

Documentation accumulates deep links to dashboards, internal service URLs, and repo paths, and every rename quietly kills a few. A dead link is drift in its most measurable form: binary, machine-checkable, and directly felt during incidents when a runbook points at a dashboard that moved. A link checker such as lychee runs over a docs tree in one command and slots into CI; internal links behind auth need the checker to run inside the network or carry a token, which is worth the setup for runbook paths specifically.

Track dead links per page and fail the docs build only on new breakage, so the backlog does not block the pipeline while it is being worked down.

# Dead links across the docs tree (lychee, single command)
lychee --no-progress docs/

# CI-friendly: fail only on newly introduced breakage
lychee --no-progress docs/ --format json > links-now.json
# diff against the stored baseline in your pipeline

Some documentation makes countable claims, and counts can be compared mechanically. The classic data-team case: the orders table has 31 columns in information_schema and 24 rows in its data dictionary page, so seven columns are undocumented and the dictionary has drifted. The same pattern applies to services in the catalog versus services with an architecture page, endpoints in the OpenAPI spec versus endpoints in the reference, and queues in infra code versus queues on the platform diagram.

Count mismatches are the gentlest drift metric to socialize because they carry no blame, only a number and a diff, and they decompose directly into a to-do list.

-- Columns the schema has vs rows the data dictionary documents
SELECT COUNT(*) AS actual_columns
FROM information_schema.columns
WHERE table_schema = 'analytics'
  AND table_name = 'orders';
-- Compare with the documented-column count for the same table;
-- a mismatch is a drift item with a ready-made worklist.

A diagram's drift check is a set comparison: nodes depicted versus services deployed, edges drawn versus dependencies in the infra code. Scripting that comparison against a hand-drawn image is impractical, which is why diagram drift is usually caught by a human squinting, late. The techniques that do exist are collected in how to detect stale diagrams.

The structural alternative is making the comparison the update: an agent connected to the Datadef MCP server reads the repository and redraws the diagram to match, so depicted-versus-deployed converges every time the loop runs, one command after a merge or in CI, never by automatic repo watching. For diagrams, regeneration is the drift metric that fixes what it measures.

Whichever checks you adopt, report them next to freshness as one score, the shape described in how to measure documentation health: freshness flags the suspects cheaply, drift checks convict precisely.

• Documentation drift
• Documentation freshness metrics
• How to detect stale diagrams
• How to measure documentation health