Data Pipeline Guide

Sources and sinks on a pipeline diagram: upstream, downstream, and who owns each end

By the engineer who builds Datadef, from client work on data platforms · Reviewed August 21, 2026

Upstream and downstream are the two words most likely to be used confidently and understood differently in the same meeting. On a pipeline, upstream means closer to where the data was created and downstream means closer to where it is consumed, so a broken upstream source breaks everything downstream of it and never the reverse. A diagram that gets the two ends right settles the vocabulary, and it also settles who gets paged.

Try it free
6 min readFor teams whose incidents start in systems they do not own

See it as a diagram

Everything below, as a diagram you can edit. Describe yours and see it in seconds.

192/2000• 3 credits left
Try:

No account needed · Editable canvas, not a picture

Upstream and downstream, defined on the picture

Upstream is toward creation, downstream is toward consumption. The application database that writes an order row is upstream of the raw table, which is upstream of the revenue mart, which is upstream of the dashboard. Failures travel downstream and blame travels upstream, which is why the direction convention on a diagram is not cosmetic.

Fix the direction and never break it. Left to right for the main flow, top to bottom only if the canvas forces it, and no back edges except for genuinely bidirectional flows such as a reverse ETL sync writing back into the CRM that is also a source. When you do draw a back edge, label it, because an unlabelled arrow returning leftwards reads as a mistake.

Mark the ownership boundary explicitly. Draw a zone around the systems your team controls, and leave the sources outside it. Everything outside that zone can change schema without warning you, and the boundary crossings are exactly the places where a data contract earns its keep.

Sources: name the system of record and the extraction method

A source node labelled with a vendor logo tells a reader almost nothing. A source node labelled Salesforce, Opportunity and Account objects, bulk API nightly tells them what data, by what mechanism, how often, and implicitly who to call. The extraction method matters because it determines the failure mode: an API extract fails on rate limits and auth expiry, a CDC reader fails on replication slot growth, a file drop fails silently by simply not arriving.

Add the schema owner where one exists. It is the single most useful annotation on a source, because the most expensive pipeline incidents are schema changes nobody announced. If the owner is a team rather than a person, name the team.

When a source produces many similar objects, collapse them. Sixty Salesforce objects should be one node reading 60 Salesforce objects, not sixty boxes. The full list belongs in the written doc next to the diagram, where it can be searched.

Sinks: every one needs a named consumer

The rule is simple and uncomfortable: if you cannot name a consumer for a sink, it is a candidate for deletion. Drawing the sinks with consumer names attached is the cheapest inventory of what your platform is actually for, and it routinely finds two or three tables built for a project that ended.

Annotate each sink with the freshness the consumer expects, not the freshness the pipeline happens to deliver. Data as of 06:00 UTC daily is a commitment. Written on the diagram, it becomes the thing everyone points at when someone asks for real time and means twice a day.

Handle fan out with a count. One gold table feeding a dozen dashboards is one node reading 12 dashboards, one arrow, and a list in the doc. Twelve arrows from one node communicate nothing except the number twelve, and they wreck the layout. When someone genuinely needs to see which column feeds which report, that is a different zoom level, which is what column level lineage is for.

Sources (outside our ownership zone)
  Salesforce      Opportunity, Account       bulk API, nightly 01:00   owner: RevOps
  Postgres app db 14 tables                  CDC, continuous           owner: Platform
  Partner SFTP    daily settlement file      file drop, ~06:30         owner: Finance Ops

Sinks (inside our zone)
  gold.revenue_daily     -> 12 Looker dashboards      freshness: 06:00 UTC daily
  gold.customer_360      -> Hightouch sync to CRM     freshness: hourly
  features.churn_v3      -> churn model training      freshness: weekly, Sunday

The boundary is the incident surface

Everything drawn outside your ownership zone can break you without asking. Keeping that zone accurate is the highest value part of the diagram to maintain. See keeping a pipeline diagram accurate.

Take the sink list from the project rather than from memory

The rule that every sink needs a named consumer holds up much better when the names come out of a file. In a dbt project the mechanism already exists: an exposure declares a downstream use of the project, with a required type of dashboard, notebook, analysis, ml, or application, a required owner carrying a name or an email, and a depends_on list that accepts ref, source, and metric references.

Two of the optional fields earn a place on the canvas. A url turns the sink node into something a reader can open during an incident instead of searching for. A maturity of high, medium, or low is a ready-made annotation for how much anyone should rely on the thing: a mart whose only exposure is marked low is a different object from one feeding a high maturity finance board, and that difference deserves to be visible.

The reverse reading is the one that pays. Any mart with no exposure pointing at it either has an undeclared consumer, which is a documentation gap you have just found, or no consumer at all, which is a deletion candidate. Running that comparison before you draw usually shortens the right hand side of the diagram, and shortens the warehouse bill along with it.

FAQ

What do upstream and downstream mean in a data pipeline?

Upstream means closer to where the data is created and downstream means closer to where it is consumed. An application database is upstream of a raw table, which is upstream of a mart, which is upstream of a dashboard. Failures propagate downstream, so an upstream schema change breaks consumers but a broken dashboard cannot break the source.

What is a sink in a data pipeline?

A sink is a destination where the pipeline delivers its output: a serving table, a dashboard, a reverse sync into an operational system, a feature store, or an API. A useful diagram gives every sink a named consumer and a stated freshness expectation, because a sink with no named consumer is usually dead weight paying for storage and compute.

How should I label a source node?

With the system of record, the specific objects or tables, the extraction method, the cadence, and the schema owner. The extraction method matters because it predicts the failure mode: API extracts fail on rate limits and expired credentials, CDC readers fail on replication slot growth, and file drops fail silently by not arriving at all.

How do I draw a table that feeds a dozen dashboards?

As one node, one arrow, and a count such as 12 dashboards, with the list kept in the written documentation. A dozen separate arrows from one node adds no information beyond the number and destroys the layout. Use column level lineage when someone needs to trace a specific field to a specific report.

Where should the ownership boundary go on the diagram?

Draw a zone around the systems your team controls and leave upstream sources outside it. That boundary is your incident surface: anything outside can change schema or stop delivering without consulting you, and each crossing of the boundary is where a data contract belongs.