Databricks Guide

Diagram a Databricks repository: the workspace your code provisions

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

A Databricks repo is two things stacked. Underneath, the platform: a workspace, a metastore, catalogs and schemas, external locations, access connectors, SQL warehouses, and job clusters, nearly always declared in Terraform. On top, the work: notebooks and SQL that run on that platform. The sync draws the first layer with real precision, and is explicit about how much of the second it can see.

Try it free
7 min readFor platform teams who provision Databricks through Terraform

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

174/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

The Terraform layer is where the architecture lives

When the repository is Terraform-dominant, Datadef does not ask what the diagram should be about. Dominant is a measured threshold rather than a feeling: at least two .tf files, and either five or more of them making up at least 30 percent of the classified files, or a 60 percent share outright. Above that line the repository is parsed rather than sampled, every .tf and .tfvars file, into an analysis of stacks, modules, resources, count and for_each shapes, and cross-module references resolved through outputs. No terraform init, no plan, no state file, and no Databricks or cloud credentials are involved.

For a Databricks platform that produces the picture people keep asking for. The workspace, the access connector and its managed identity, the storage account behind the external locations, the metastore, the catalogs, the SQL warehouses, and the job compute all render as nodes with their real provider icons inside module zones named the way your repo names them.

Identity wiring is counted rather than drawn. databricks_grant, databricks_grants, databricks_permissions, databricks_service_principal, databricks_group_member and their relatives are classified as wiring, along with anything whose type name carries role, permission, grant, member, iam, entitlement or acl. On a Unity Catalog repo they routinely outnumber the platform itself, so the diagram carries a line of the form "38 wiring resources not worth boxes: 14 grants, 12 permissions, 8 role assignments" instead of thirty-eight rectangles. The full inventory stays in the generated module reference table.

The curation is deterministic, decided in code rather than left to the model. Repeated minor types inside a module collapse into one node carrying the count and the member names, in the shape "Storage credentials ×2: external, root". A module that still overflows gets a "Supporting resources ×N" node holding the tail. The whole plan aims at about forty nodes, no more than eight per module zone, with a deeper budget of sixteen when one module is more than half the repository and a wider total for a catalog-shaped repo of many small stacks. A 123-resource platform lands at roughly 36 drawn nodes, and the header of the analysis states both numbers so nothing is quietly lost.

Environments come out as counts, not as three diagrams

Databricks platform repos are usually multi-environment: one module set, several tfvars files, different catalog counts per environment. Environments are recognized from the layout, any path segment matching environments/<NAME>/ or envs/<NAME>/, and variables are followed through call sites, defaults, and per-environment tfvars. A catalog resource created per project therefore appears as one node whose description reads "per lakebase_projects (DEV 10, QUAL 5, PROD 1)" rather than as three cloned subtrees. A count of zero reads as "not deployed (count 0)", and a conditional the parse cannot resolve is quoted verbatim, so a reader sees count: local.is_dev ? 1 : 0 and understands it instantly.

That is a specific choice against two common failure modes. Drawing three cloned platforms triples a diagram that is structurally identical, and writing a generic times-N label throws away the information a reviewer actually wants. Stating counts on one node keeps both the shape and the facts.

Registry and git modules cannot be expanded from your repository alone, so each call is drawn as one component carrying its name, source, and version, wired through its input references. A repo composed largely of shared Databricks modules still produces a real diagram rather than a single opaque box.

Detailed by design

Most diagrams are generated against an overview budget of roughly 12 to 20 nodes. A parsed Terraform repository is generated at the detailed scope instead, because the maintainer has to find every module they own, which is why a platform repo lands near forty nodes rather than fifteen. See repository sync documentation.

What the notebook side contributes

Notebooks exported as .ipynb are on the binary skip list and are never fetched, and notebook source saved as Python modules is not part of the classified corpus either. What the generator sees of them is the directory summary: notebooks/ (61 files), src/jobs/ (18 files), and so on, to three directory levels with counts.

SQL is different. Any .sql file in the repository is classified, and .sql under a models/ directory gets its own class, so a repo that keeps its transformations as SQL contributes them to the corpus and to the diagram. A repo that keeps everything in notebooks contributes structure but not content.

The practical consequence: for the medallion layers themselves, either keep the transformations as SQL or draw the layer diagram from a description and edit it. For the platform underneath, the Terraform parse is authoritative and needs nothing from you but read access.

From connection to a diagram that keeps up

Connect the repository on GitHub, GitLab, or Azure DevOps with a read-only token, choose a branch or tag, and the first diagram streams onto the canvas within a couple of minutes. The daily sync re-reads the same ref, skips unchanged commits, and preserves the positions of any nodes you moved by hand.

Alongside the diagram the sync writes an architecture doc covering what the platform provisions, how environments differ, and how the modules wire together, with a generated module reference table appended. Embed the diagram in the repo README or a Confluence page as a live image and the platform page stops aging.

FAQ

Does this connect to my Databricks workspace?

No. The analysis is a parse of the repository source. There is no workspace API call, no personal access token for Databricks, no cloud credentials, and no Terraform state access. Read access to the repository is the whole requirement.

Are my notebooks read?

Files with the .ipynb extension are skipped as binary, and notebook source stored as Python modules is not classified either. The repository tree summary still reports the notebook directories and their file counts, so the generator knows they exist and how large they are.

How are Unity Catalog grants and permissions drawn?

They are not drawn as boxes. Grants, role assignments, and permission resources are classified as identity wiring and summarized in a note with a count and what they secure, because on a real Unity Catalog repo they would otherwise outnumber the platform itself.

We deploy the same modules to DEV, QUAL, and PROD. Do we get three diagrams?

One diagram with honest counts. Environments are read from the environments or envs directory layout, and the tfvars in each are parsed, so a resource created per project stays a single node whose description names the variable that drives it and the real number in each environment. Drawing three near-identical platforms would triple the diagram without adding a fact, and a generic times-N label would throw away the facts a reviewer wants.

What if the repo has no Terraform at all?

Then it is treated like any other repository: the selector takes the containers, deployment manifests, SQL, CI pipelines, and dependency manifests it finds, up to 40 files and 250KB, and the first sync proposes a focus based on what those counts show. The same applies below the Terraform threshold, so a repo with one .tf file beside a large notebook tree goes down the sampling path rather than the parsing one.