Docs Automation Guide

Documentation review in a pull request: the checkbox that never worked

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Every pull request template has a line that says documentation updated, and on most teams it is ticked without being read. That is not a discipline problem. The checkbox asks the wrong person, at the wrong moment, a question they cannot verify from the diff, and it asks it identically on a typo fix and on a new service.

6 min readFor teams whose PR template has a docs checkbox nobody trusts

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

164/20003 credits left
Try:

No account needed · Editable canvas, not a picture

Why the checkbox fails, mechanically

It is self-attested. The person who ticks it is the person who wants the pull request merged, at the moment they want it merged, which is the worst possible combination of incentives and timing in the whole workflow.

It is unverifiable from the diff. A reviewer looking at a code change cannot see whether a runbook in a wiki, an onboarding page in Notion, or an architecture diagram somewhere else is now wrong. Asking them to confirm it is asking them to guess, and reviewers who are asked to guess learn to approve.

And it is uniform. The same line appears on a dependency bump and on a change that introduces a queue. A control that fires identically on everything trains people to ignore it, which then costs you the few cases where it mattered.

Route by path instead of by checkbox

A CODEOWNERS entry on documentation paths puts a named human on the review automatically, and a required review has consequences a checkbox does not. The important part is scoping: apply it to the paths where staleness is expensive, typically runbooks, onboarding material, and the architecture directory, and leave the rest alone.

Ownership by path also fixes the diffusion problem. Documentation that belongs to everyone belongs to nobody, and the usual result is a wiki with four contradictory pages on the same subject. A path with an owner has one person who notices.

Keep the routing narrow enough that the owner can actually read what lands on them. A docs owner receiving every pull request in the repository is the checkbox again, with a person attached.

# .github/CODEOWNERS
# narrow on purpose: only the paths where staleness costs something

/docs/runbooks/         @acme/sre
/docs/onboarding.md     @acme/platform-leads
/docs/architecture/     @acme/architecture

# deliberately absent: a catch-all owner on every path
# *                     @acme/docs

Ask three questions the diff can answer

Replace documentation updated with three specific questions, each tied to an artifact. Does this pull request add or remove a deployable unit. Does it change what talks to what. Does it change an operational procedure someone follows during an incident.

The first two point at the architecture diagram and the component inventory. The third points at a runbook, and it is the only one of the three that genuinely needs a human to write the answer. A pull request that answers no to all three needs no documentation review at all, which is the real objective: making the control meaningful by making it rare.

These questions have another property the checkbox lacks. A reviewer can answer them from the diff, because adding a service, changing a connection, or editing a deployment procedure all show up in the changed files.

Regenerate the parts a human should not be reviewing

The component inventory, the data flow description, the deployment section, the directory notes, and the diagram are all derivable from the branch. Reviewing them by hand is asking a person to do a diff that a machine does better, and the person will do it worse every week until they stop.

A repository connection regenerates both artifacts on the tracked branch, skipping the work entirely when the head commit has not moved. For teams that want the refresh tied to the merge rather than to a schedule, two MCP tools do it: repo_status reports which repository and ref a diagram tracks, when it last synced, the commit it reflects, and whether daily sync is on, and repo_refresh re-syncs it. Both take a project_id, and both are cheap to call from a merge hook, because a refresh on a commit whose structure did not change stops before any generation runs.

What is left for human review is the part that is actually judgment: the reasoning, the procedures, the warnings, the parts of the system nobody would guess. That review is worth requesting, worth blocking on, and short enough that people do it.

FAQ

Why do documentation checkboxes in PR templates not work?

They are self-attested by the person who wants the merge, they ask about artifacts a reviewer cannot see in the diff, and they appear identically on a typo fix and on a new service. A control that fires on everything and can be answered without checking anything trains people to tick it automatically.

What should replace the docs checkbox?

Three questions a reviewer can answer from the diff: does this change add or remove a deployable unit, does it change what talks to what, and does it change an operational procedure. Each maps to a specific artifact, and a pull request answering no to all three needs no documentation review.

How do CODEOWNERS help with documentation review?

They route review by path rather than by self-assessment, so a change touching runbooks, onboarding material, or the architecture directory automatically requires a named reviewer. Scope it narrowly, because an owner who receives every pull request is a checkbox with a person attached.

Which documentation should never be reviewed by hand?

The parts derivable from the branch: the component inventory, the data flow description, the deployment section, directory notes, and the architecture diagram. Regenerating them is more reliable than asking a reviewer to check them, and it frees review time for reasoning and procedures.

Can documentation be regenerated on merge instead of on a schedule?

Yes. Alongside the daily sync, two MCP tools handle it: repo_status reports the tracked ref, the last synced commit, and whether daily sync is on, and repo_refresh re-syncs the project. Both take a project_id. A refresh on a commit whose structure has not changed stops before generating anything, so triggering on every merge costs close to nothing.