GitLab CI pipeline diagram: resolve the includes, diagram the needs DAG

GitLab will do the merge for you: the CI Lint API validates the configuration in the context of the project and returns the fully merged YAML, includes resolved, in the merged_yaml field of the response. The pipeline editor shows the same thing interactively in its full configuration tab. Either way, the merged document is the diagram input, because it is the only artifact in which a job contributed by a template three includes deep actually appears.

# The merged, effective config: includes resolved by GitLab itself
curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  "$CI_API_V4_URL/projects/$CI_PROJECT_ID/ci/lint" \
  | jq -r '.merged_yaml'

Without needs, GitLab runs stages as a strict sequence: every job in test waits for all of build, whether it depends on it or not. needs breaks that: a job with needs: [compile] starts the moment compile finishes, regardless of stage, and a job with needs: [] starts immediately at pipeline creation. The real execution graph is the needs DAG, and the stages are just the columns it is drawn over.

That split is exactly how to draw it: stages as background columns, needs as explicit arrows cutting across them. The gaps become visible in a way YAML never shows: the job still waiting on an entire stage when it needs one artifact, the needs: [] utility job nobody realizes runs first, the fan-in job that serializes an otherwise parallel pipeline.

trigger with include spawns a child pipeline inside the same project, and the config can even be generated by an earlier job and passed as an artifact, a dynamic child pipeline whose shape is decided at run time. trigger with project starts a pipeline in a different project entirely. Both belong on the map as subgraphs with a trigger edge in: collapsed to a single node when the reader only needs to know the handoff exists, expanded when the child is the pipeline being explained.

Dynamic children deserve an honest annotation: their merged config does not exist until a run generates it, so the standing diagram shows the generator job and a representative child, not a guaranteed shape.

The refresh belongs in the pipeline it documents: a job guarded by rules with changes on the CI config paths runs only when the pipeline definition itself changes, pulls the merged YAML from the Lint API, and hands it to an agent connected to Datadef's MCP server, registry name io.datadef/mcp. Stated plainly: Datadef does not watch the repository, and GitLab does not call Datadef; the rules:changes guard is the detection and the agent invocation is the one command, authenticated by an API key kept in CI variables, available on paid plans.

First-time generation is a different job than refreshing: paste the merged YAML or a description into the DevOps pipeline diagram generator and shape the initial map there, then let the pipeline own every update after.

refresh-pipeline-diagram:
  stage: .post
  rules:
    - changes:
        paths:
          - ".gitlab-ci.yml"
          - "ci/**/*.yml"
  script:
    - MERGED=$(curl --silent --header "PRIVATE-TOKEN: $GITLAB_TOKEN"
      "$CI_API_V4_URL/projects/$CI_PROJECT_ID/ci/lint" | jq -r '.merged_yaml')
    - echo "$MERGED" > merged-ci.yml
    # Hand merged-ci.yml to your MCP-connected agent to update the diagram

The merged YAML is the potential pipeline, not any single run: rules decide per pipeline which jobs exist, so a job gated on a schedule or a variable may never appear in the pipelines a given reader sees. The diagram should show the full graph and mark condition-gated jobs as such, rather than pretending every run looks the same.

And the standing constraints: embedding the live diagram in a GitLab wiki or README, covered in embedding diagrams in GitLab, requires the Datadef project shared public, and the loop keeps the pipeline map current, not the runbook prose around it.

• DevOps pipeline diagram generator
• A living diagram from GitHub Actions
• Embed diagrams in GitLab
• The Datadef MCP diagram server