DATABRICKS DATA QUALITY MONITORING
Databricks Data Quality Monitoring: Data Quality Checks, Expectations, and Anomaly Detection in Databricks
Data quality monitoring across every Delta table, watching freshness, volume, schema, and distribution, with no monitor to configure per table and no rule to author per column. Read only connection, alerts in Slack and PagerDuty, from 99 dollars a month.
14-day trial, no credit card, read-only connection
Alerted #data-eng 0.8s ago.
Downstream impact · consumers at risk
How do you monitor data quality in Databricks?
Databricks gives you four native routes and they solve different problems. Unity Catalog anomaly detection is the one click option: enable it on a schema and it models each table commit history and row counts to flag stale and incomplete tables, writing results to the system.data_quality_monitoring.table_results system table. Data profiling, the feature formerly called Lakehouse Monitoring, computes summary statistics and drift metrics on tables you configure individually. Lakeflow Declarative Pipelines expectations enforce row level rules, but only inside a pipeline. DQX, a Databricks Labs framework, adds quality checks to PySpark code you run yourself. None of them cover schema drift on tables outside a pipeline, distribution monitoring on columns you did not nominate, or alert routing beyond workspace email. Dataobservability connects to Databricks read only, learns each Delta table baseline from history, and alerts on all four pillars across every catalog, from 99 dollars a month.
Last updated August 2026
Side by side
Databricks data quality monitoring compared
Swipe to see all columns →
| How to monitor data quality in Databricks | What it catches | What you build and operate | Documented limits (Databricks docs, August 2026) |
|---|---|---|---|
| Unity Catalog anomaly detection | Freshness and completeness on every table in an enabled schema, plus percent null in beta | Enabling it on a schema, then somewhere to send results beyond workspace email | Does not support views or foreign tables. Completeness does not account for the fraction of nulls, zero values, or NaN. Intelligent scanning can delay health indicators by up to two weeks for a table it chose to skip |
| Data profiling, formerly Lakehouse Monitoring | Summary statistics and drift metrics on the tables you configure, with a generated dashboard | One monitor per table, a refresh schedule, and something that reads the generated metric tables | Snapshot profiles cap at a 4TB table size. Time series and inference profiles only compute metrics over the last 30 days, and on creation analyze only the 30 days prior. Notifications support at most 5 emails per event type |
| Lakeflow Declarative Pipelines expectations, formerly DLT expectations | Row level rules you wrote, enforced as records are written, with warn, drop, or fail behavior | Every rule, and a decision per rule about whether a bad record warns, gets dropped, or fails the update | Work only inside pipelines, on streaming tables, materialized views, and temporary views. Constraints cannot contain custom Python functions, external service calls, or subqueries. Not supported with AUTO CDC FROM SNAPSHOT |
| DQX, the Databricks Labs quality framework | Row and column level checks on PySpark DataFrames, with quarantine, severity levels, and generated rule candidates | Installation, a checks config, the job that runs it, and the dashboards on top | A Databricks Labs project rather than a supported Databricks product, so upgrades and failures are yours. Operates on PySpark DataFrames, so tables written by anything outside that path are not covered |
| Great Expectations or hand written SQL checks | Anything you can express as an assertion about a column | The suite, the runner, the schedule, the results store, the deduplication, and the alerting | Every check is a statement one person wrote about one column on one day, and the warehouse changes faster than the check set gets revisited |
| Dataobservability | Freshness, volume, schema, and distribution breaks across every Delta table, with lineage and grouped incidents | A read only connection, no agent | Covers the connected warehouses (Snowflake, BigQuery, Databricks, Redshift) and dbt. Systems outside them are not monitored |
Positioning and pricing models are summarized in good faith from each vendor's public pages, August 2026. Verify current terms with the vendor.
What you get
What a monitoring layer adds on top of the Databricks primitives
Coverage that does not depend on which tables you nominated
Data profiling is configured one table at a time. That is the right shape when you have a dozen tables that carry the business and an engineer who knows which they are. Across a catalog of a few thousand Delta tables it means the quality of your monitoring equals the quality of somebody memory on the day they set it up, and the table nobody thought about is exactly the one that breaks. Metadata driven monitoring inverts the default: the table list is the input, so a new table arrives already watched.
All four pillars, not just freshness and completeness
Unity Catalog anomaly detection genuinely covers two things well, and Databricks is clear about which two: freshness from commit history and completeness from row counts, with percent null still in beta. What it does not watch is schema drift on tables outside a pipeline, and distribution shifts on columns nobody nominated in advance. Those are the failure modes that reach a dashboard silently, because a retyped column or a currency field that switched units does not change the row count at all.
Alerts that reach the engineer instead of an inbox
Native alerting notifies selected workspace users by email, and data profiling notifications are capped at five email addresses per event type. That is a notification, not an on call path. Getting from there to an engineer who acts means an ownership map, deduplication so the same stale table does not mail you every scan, and an integration with the tool your team already watches. Routing per dataset owner into Slack and PagerDuty is the difference between a signal and a filter rule.
Baselines that already know your table is quiet on Sunday
Most false alarms are calendar failures rather than statistical ones. A table loaded hourly on weekdays and never at the weekend is not broken on Saturday, and a fixed threshold cannot tell the difference. Baselines derived from each Delta table own commit history encode weekday patterns, month end spikes, and the batch that lands at 04:15, without anyone writing an exception for any of them.
Freshness that reads the operation, not just the timestamp
Delta writes a new table version for operations that change nothing about the contents. An OPTIMIZE compaction, a VACUUM, or a table property change all move the last commit timestamp forward. A freshness check built on commit recency therefore reports a table as fresh while its upstream job has been failing for two days, which is how a stale dashboard survives an entire reporting cycle. Monitoring that distinguishes the operation type does not have that blind spot.
A cost line that does not scale with row scans
Native quality monitoring runs on serverless compute and is billed as serverless DBUs under the DATA_QUALITY_MONITORING billing product, with cost scaling by the number and size of tables monitored. The fastest way to make that number uncomfortable is a tool that profiles rows on every cycle. Reading Unity Catalog metadata, Delta commit history, and file statistics first, and sampling rows only when a metric genuinely needs them, is what makes full catalog coverage affordable rather than a pilot on twelve tables.
How it works
From connected to caught
Connect Databricks with a read only service principal
Create a service principal with USE CATALOG and SELECT on the schemas you want watched, plus access to Unity Catalog metadata. Dataobservability reads catalog metadata, Delta transaction history, and table statistics to build the inventory and the freshness and volume picture, so table contents are not copied out of your workspace. There is no agent to install, nothing running inside your clusters, and no change to existing jobs or pipelines.
Let baselines build before you touch a threshold
Give it a few days of commit history per table so weekly seasonality becomes visible. Tuning thresholds before a baseline exists is how teams manufacture alert fatigue in the first week and stop trusting the monitoring by the second. The right amount of configuration on day one is close to none, which is also the argument for enabling Unity Catalog anomaly detection at the same time and comparing what each one flags.
Keep expectations for the invariants, not the general shape
Lakeflow expectations are good at what learned baselines cannot know: that a discount percentage must never exceed 40, that every order carries a valid region code, that a foreign key resolves. Use them there, with fail on the handful of rules where a bad record must never land, and warn on the rest. Leave drift, outages, and distribution shifts to baselines. That split is what keeps the rule set finite instead of growing until nobody dares delete anything.
Route by dataset owner, then prune what never fires
Send alerts to the team that owns each catalog or schema rather than into one shared channel. After two weeks, read the log back: a monitor that has never fired on a real problem should be relaxed, and any incident that reached you through a complaining stakeholder should become a monitor. Monitoring that is never pruned turns into noise, and noise gets muted, usually by the person you most needed to reach.
Unity Catalog anomaly detection, and the two questions it answers
This is the newest and by some distance the most useful of the native routes, because it is the only one that gives coverage without per table configuration. You enable it on a schema, which needs the MANAGE privilege on that schema or its parent catalog, and Databricks starts building a model per table from history. Freshness works by analyzing the history of commits to a table and predicting when the next one should arrive, marking the table stale when a commit is unusually late. Completeness analyzes historical row counts, predicts a range for the number of rows expected in a 24 hour window, and marks a table incomplete when the actual count falls below the lower bound of that range. A percent null check over the last 24 hours is documented as beta. Results land in the system.data_quality_monitoring.table_results system table, and there is real root cause work on top: for each unhealthy table it uses Unity Catalog lineage to reason about multi level relationships and decide whether the problem started in that table or in something upstream. All of that is genuinely good, and if you run Databricks you should turn it on. The boundaries are equally documented. It does not support views or foreign tables, so anything your analysts actually query through a view sits outside it. Completeness explicitly does not account for the fraction of nulls, zero values, or NaN, which means a load that arrives with the right row count and an empty column reads as healthy. And scanning is deliberately uneven: Databricks scans each table at roughly the frequency it is updated, prioritizes tables by popularity and downstream usage, and reduces frequency for less critical ones. The documented consequence is that health indicators for a skipped table can be delayed by up to two weeks. That is a sensible cost control and a real gap, because the table nobody queries is often the one feeding the report that only matters at quarter end.
Data profiling, the feature you probably still call Lakehouse Monitoring
Lakehouse Monitoring reached general availability in August 2024 and now appears as data profiling inside the Unity Catalog data quality monitoring suite. It answers a different question from anomaly detection: not whether a table is broken, but what its data looks like and how that shape is moving. You create a monitor on a specific table and pick one of three analysis types. Time series is for tables built on a timestamp column. Inference is for tables holding a model request log, which is where the machine learning monitoring story lives. Snapshot is for everything else. Databricks then generates two Delta tables, a profile metric table carrying summary statistics and a drift metrics table tracking how those statistics change, and builds a dashboard over them. It runs on serverless compute for jobs, and notably does not require your account to be enabled for serverless compute generally. Four limits shape how far this goes. A snapshot profile caps at a 4TB table size, and Databricks recommends time series profiles above that. Time series and inference profiles only compute metrics over the last 30 days, and when first created they analyze only the 30 days prior to creation, so there is no deep backfill of history you already have. Monitors defined on materialized views do not support incremental processing. And notifications support at most five email addresses per event type. Under those constraints sits the same structural cost as any per object configuration: one monitor per table, created by a person who decided that table mattered. Two hundred important tables is two hundred monitors, each with a refresh schedule, each generating two metric tables, and none of them covering the table added last Tuesday.
Expectations and DQX: two ways to write rules, both of which you operate
Lakeflow Declarative Pipelines expectations, which most teams still call DLT expectations, are the cleanest rule mechanism Databricks ships. An expectation has three parts: a name for tracking, a constraint clause that is a SQL conditional evaluating to true or false per record, and an action on violation. The three actions are worth memorizing because the default surprises people. Warn is the default and invalid records are still written to the target, they are simply counted. Drop removes invalid records before the write. Fail stops the update and requires manual intervention. In Python that is the expect, expect_or_drop, and expect_or_fail decorators; in SQL it is CONSTRAINT name EXPECT condition with ON VIOLATION DROP ROW or ON VIOLATION FAIL UPDATE. Metrics surface in the pipeline Data quality tab, or from the event log for standalone SQL pipelines. The constraints are documented and consequential. Expectations work only within pipelines, on streaming tables, materialized views, and temporary views, so a table written by a notebook or a job outside a pipeline cannot have one. A constraint cannot contain custom Python functions, external service calls, or subqueries, which rules out referential checks against another table and any validation needing a lookup. And they are not supported with AUTO CDC FROM SNAPSHOT. DQX is the Databricks Labs answer to some of that. It is a data quality framework for Apache Spark that defines checks as code or config, applies row and column level rules to PySpark DataFrames in batch or structured streaming, supports warning and error severities, and can drop, mark, or quarantine failing records rather than only failing the job. It profiles data to generate rule candidates, can build checks from ODCS data contracts, and ships a browser UI and quality dashboards. It is also, importantly, a Labs project rather than a supported Databricks product, and it operates on DataFrames, which means it covers the code you route through it and nothing else. Both are rule engines. Neither is a monitoring system, because neither decides what normal looks like for a table you never wrote a rule about.
Where the four native routes leave a gap, stated plainly
Line the four up against the pillars and the picture is consistent. Freshness is covered well by anomaly detection, with a per table learned model, which is exactly right. Volume is covered for the 24 hour window, with the caveat that completeness ignores nulls, zeros, and NaN, so a full but empty column passes. Distribution is covered only where you created a data profiling monitor and only for the last 30 days of metrics. Schema drift is the weakest: expectations catch a rule you wrote about a column you knew about, and nothing native watches a catalog for columns being added, dropped, or retyped underneath you. Lineage is strong inside Databricks and stops at the workspace edge, which matters because the damage from a bad table is almost always noticed in a BI tool rather than in a notebook. Then there is the operational layer, which is where most of the real effort goes and which none of the four provide. There is no deduplication, so a table stale for three days produces an event on every scan. There is no ownership model, so alerts go to whoever enabled the feature rather than to whoever owns the dataset. There is no acknowledgement or resolution state, so nobody can tell whether an incident is being handled. Alerting is email to selected workspace users, capped at five addresses per event type on the profiling side. And there is no monitoring of the monitoring, so a scan that silently stops running looks identical to a warehouse where nothing is wrong. Every one of those is buildable. Together they are a quarter of an engineer, permanently, and that is the honest comparison to run against a subscription.
Build on the primitives or buy the layer above them
The straight answer is that native Databricks quality monitoring is good enough for a real set of teams, and if you are one of them you should not pay anyone. If you run a few dozen tables that matter, most of them inside pipelines you control, with an engineer who is happy owning expectations and a couple of profiling monitors, turn on anomaly detection, write your expectations, and stop reading. The line moves when coverage becomes the goal rather than a curated list. At that point you are not writing checks any more, you are operating a system: schedules, a state store so repeat problems do not re alert, per table thresholds someone maintains, an ownership map, routing into Slack or PagerDuty, acknowledgement and resolution, backfill for the week after an incident when somebody asks how long it had been running, and monitoring of the monitoring. That work is not difficult, which is exactly why it gets underestimated. Teams ship version one in an afternoon and spend two quarters on the parts nobody scoped, funded by the most senior data engineer available and taken straight out of the pipeline work they were hired for. It is also why published pricing matters more in this category than in most. Of the thirteen observability tools we track, three publish a price: Soda at 750 dollars a month for its Team plan, Metaplane, and us. Monte Carlo publishes no list price on its own site, though its AWS Marketplace listing shows a 12 month contract at 50,000 dollars for its credit dimension, with overage at one cent per unit. Bigeye pricing page returns a 404, while its AWS Marketplace listing shows 45,000 dollars a year for 100 actively monitored tables and 75,000 dollars for 300. Datafold redirects its pricing page to a contact form. All three figures were checked directly on the listings in August 2026. Dataobservability is 99, 299, and 799 dollars a month, with a 14 day trial that needs no credit card, so you can hold a real number against the engineering estimate before anyone books a call.
Questions buyers ask
Databricks data quality monitoring FAQ
How do you monitor data quality in Databricks?
Enable Unity Catalog anomaly detection on the schemas that matter, which gives you freshness and completeness on every table without per table setup. Add data profiling monitors on the specific tables whose distributions you need to watch. Use Lakeflow expectations for row level invariants inside pipelines. Then add a monitoring layer for schema drift, distribution across the whole catalog, deduplication, ownership, and alert routing, none of which the native features provide.
What is DQX in Databricks?
DQX is a data quality framework for Apache Spark from Databricks Labs. It applies row and column level checks to PySpark DataFrames in batch or structured streaming, defined as code or config, with warning and error severities and the option to drop, mark, or quarantine failing records. It also profiles data to generate candidate rules and can build checks from ODCS data contracts. Being a Labs project, you run and support it yourself.
What are expectations in Databricks?
An expectation is a data quality constraint inside a Lakeflow Declarative Pipeline, previously called Delta Live Tables. It has a name, a SQL conditional that must evaluate true or false per record, and an action on violation. Warn is the default and still writes invalid records to the target. Drop removes them before the write. Fail stops the update and needs manual intervention.
Do Databricks expectations work outside a pipeline?
No. Expectations are only available within pipelines, applied to streaming tables, materialized views, and temporary views, not to standalone queries. A Delta table written by a notebook or a plain job cannot carry one. Constraints also cannot contain custom Python functions, external service calls, or subqueries, so referential checks against another table are out of scope.
What is the difference between anomaly detection and data profiling in Databricks?
Anomaly detection is automatic and schema wide: enable it once and Databricks models each table commit history and row counts to flag stale or incomplete tables. Data profiling is manual and table specific: you create a monitor on one table, choose a time series, inference, or snapshot analysis, and Databricks generates profile and drift metric tables plus a dashboard. One gives coverage, the other gives depth on tables you nominate.
What are the limits of Databricks data profiling?
Four matter in practice. Snapshot profiles cap at a 4TB table size, above which Databricks recommends time series profiles. Time series and inference profiles compute metrics over the last 30 days only, and on creation analyze only the 30 days prior. Monitors on materialized views do not support incremental processing. Notifications support at most five email addresses per event type.
Does Databricks anomaly detection work on views?
No. The documentation states that anomaly detection does not support views or foreign tables, so it covers managed and external Delta tables only. That gap matters more than it first sounds, because views are frequently the objects analysts and BI tools query directly. A monitoring layer that treats views as first class assets closes it without you changing how your consumers access data.
What is the system.data_quality_monitoring.table_results table?
It is the Unity Catalog system table where anomaly detection writes its results, so you can query table health rather than only reading it in the UI. It is also the practical integration point if you want to route findings somewhere Databricks does not send them natively, since built in alerting notifies selected workspace users by email. Querying it on a schedule is how most teams build a first alerting bridge.
How do I get Databricks data quality alerts into Slack?
Natively you cannot, directly. Anomaly detection alert rules notify selected workspace users by email, and data profiling notifications are capped at five addresses per event type. The common workaround is a scheduled job querying system.data_quality_monitoring.table_results and posting to a webhook, which then needs its own state table so the same stale table does not post on every scan. A monitoring platform routes to Slack and PagerDuty per dataset owner with deduplication already handled.
What is the best data quality tool for Databricks?
It depends on whether you are covering a list or a catalog. For a few dozen tables inside pipelines you control, Lakeflow expectations plus Unity Catalog anomaly detection is genuinely sufficient and costs nothing extra beyond serverless DBUs. Once you need schema drift, distribution monitoring across every table, deduplicated alerts routed to owners, and incident history, a platform that learns baselines per table costs less than the engineering time it replaces.
More of the platform
Catch broken data before your stakeholders do
Connect your warehouse and get databricks data quality monitoring live from one read-only connection. Transparent pricing, no credit card.