Data Quality Best Practices: Checks, Cadence, Alerts

1. What Are the Best Practices Data Engineers Use to Manage Data Quality?

Eight practices, in the order that pays off. The first four can be in place this week; the last four are what separate a test suite people trust from one they mute. Every data team has the same horror story: a dashboard showed revenue up 40%, the CEO announced it in an all-hands, and then someone noticed a duplicate join. Data quality is not about perfection, it is about catching issues before they reach decision-makers.

What the practice buys you

2. The Six Dimensions of Data Quality (and the Test for Each)

The dimensions are the vocabulary, not the work. Their value is that each one maps to an assertion you can write in SQL tonight, so "improve data quality" becomes six specific tests rather than a quarterly objective.

3. Types of Data Quality Checks, With Examples

Six check types catch the overwhelming majority of production incidents. They are high-value and low-effort: each is one assertion against one table. If you want the argument for why these six and not four hundred, the companion guide on data quality checks that matter makes it in detail.

Two more are worth adding once those six are running: schema drift, which flags a column added, removed or retyped upstream, and referential integrity, which catches orphaned foreign keys after a join. The same six express naturally in dbt as schema tests.

# models/marts/orders.yml
version: 2
models:
  - name: orders_mart
    description: "Order-level facts for analytics"
    columns:
      - name: order_id
        tests:
          - unique
          - not_null
      - name: customer_id
        tests:
          - not_null
          - relationships:
              to: ref('dim_customers')
              field: customer_id
      - name: order_total
        tests:
          - not_null
          - dbt_utils.accepted_range:
              min_value: 0
              max_value: 1000000

4. Where Do Data Quality Checks Belong in the Pipeline?

At the boundaries: validate data when it enters your system and when it leaves. That catches both upstream issues and transformation bugs, and it keeps intermediate models out of the alert path, where a failing test on something nobody consumes is information for a pull request rather than a page. In an ETL pipeline the boundaries are the extract landing zone and the published mart; in ELT they are the raw schema and the serving layer. Deciding where checks go is much easier with a data pipeline diagram in front of you; the boundaries are obvious once drawn.

5. How Often Should Data Quality Checks Run: Daily, Weekly, or Monthly?

Cadence follows cost and purpose, not habit. Anything that can block a dashboard runs with the pipeline, on every execution, because a clock-driven check on a pipeline that runs more often than the clock serves bad data in between. A nightly sweep on an hourly pipeline means up to eleven hours of unverified data reaching consumers. Anything that compares two systems runs daily, once both sides have landed. Anything about trend runs weekly. Anything about the health of the test suite itself runs monthly.

6. How to Monitor Data Freshness and Accuracy

These two get asked together and need completely different mechanics. Freshness is measurable from inside the warehouse. Accuracy is not, because the warehouse holds no independent truth to compare against.

Write the outcome of every check to a results table: timestamp, table, check name, pass or fail, observed value. Without that history you can say whether quality is green this morning, but not whether it improved this quarter, and the second question is the one you will be asked.

7. How to Alert on Data Quality Without Alert Fatigue

The biggest risk with data quality monitoring is not missing issues, it is alert fatigue. Too many false positives and your team starts ignoring alerts, at which point the suite has negative value: it costs money and creates the belief that something is watching.

Explicit checks catch the failures you thought of. The ones you did not are caught by signals nobody wrote a test for: a table that stopped arriving, a row count that halved, a schema that gained a column overnight. That is the difference between quality testing and data observability, and the two answer different questions. If you are drawing the monitoring side of the platform, the data observability platform diagram generator lays out collectors, checks and alert routing as a starting point.

Alert Severity Framework

8. How to Expose Data Quality Metrics to Stakeholders

Nobody outside the data team reads test output, and a link to a testing tool that needs an account gets clicked once. Four numbers per data product carry the whole message, and they belong wherever stakeholders already look: the top of the dashboard, the Confluence page, the team channel.

The second thing that works is showing where the checks are. A pipeline diagram with the quality gates marked answers "is this number checked" faster than any status page, and it answers it for people who will never open the testing tool. The failure mode is the diagram itself going stale: a screenshot drawn in March describes a pipeline that has since gained two sources.

9. Which Data Quality Tools Catch Pipeline Issues Before Broken Data Reaches Dashboards?

Two categories, and the honest answer is that most teams need one from each. Testing tools check what you declared and can block a run, which is what stops bad data reaching a dashboard. Observability platforms learn the normal pattern and flag deviation nobody wrote a rule for, which is what catches the incident class you did not anticipate. Buy the second only after the first has owners, otherwise you get anomaly alerts on tables nobody is responsible for.

dbt tests or Great Expectations?

This comes up on every stack decision, and it is not either/or. dbt tests are YAML declarations living next to the model, run with dbt build, and they fail the run, which is the right fit for anything already transformed inside the warehouse. Great Expectations is a Python library whose expectation suites run against pandas, Spark or a SQL connection, so it validates files, API extracts and streams before they land, and it produces a documentation site from the suites. Teams already on dbt normally start with dbt tests, keep them for everything in the warehouse, and add Great Expectations only at the edges where data has not arrived yet.

10. How Long Does It Take to Implement a Data Quality Testing Framework?

The first useful checks take a day. A framework the team trusts takes a quarter. The gap between those two numbers is not tooling, it is ownership and alert routing, which is exactly the part that gets deferred.

11. Quality, Lineage, and Contracts: What to Put in Place Across a Platform

Checks catch the break. Three surrounding practices decide how expensive the break is, and they are what a data leader is really being asked about when the question is framed at platform scale rather than table scale.

12. Frequently Asked Questions