Databricks Data Quality Monitoring: Anomaly Detection and What It Misses

Databricks has two native ways to watch data quality, and they solve different problems. Anomaly detection is the one-click option: turn it on for a schema in Unity Catalog and it learns each table's commit pattern and row counts, then flags tables that are late or short. Lakehouse Monitoring is the deeper option: you define a monitor on a specific table and it computes profile and drift metrics on a schedule, writes them to metric tables, and builds a dashboard. Neither one covers schema drift, distribution shifts across arbitrary columns, or lineage-based impact, which is the gap most teams fill with an observability platform.

Last updated July 2026.

What is Databricks anomaly detection?

Anomaly detection is a Unity Catalog feature that monitors every table in a schema for two things: freshness and completeness. For freshness, it reads the history of commits to a table, builds a per-table model that predicts when the next commit should arrive, and marks the table stale if the commit is unusually late. For completeness, it looks at historical row counts and predicts a range of rows it expects in the last 24 hours; if the actual count falls below the lower bound of that range, the table is marked incomplete. A percent-null check on columns is also available in beta, comparing the last 24 hours of null rates against historical ranges.

The appealing part is that you do not configure per-table thresholds. Databricks uses what it calls intelligent scanning, which prioritizes tables that look important and skips low-impact ones, so enabling a schema with 800 tables does not mean paying to scan 800 tables at the same intensity. Results land in the system.data_quality_monitoring.table_results system table, which you can query like anything else, and health indicators appear next to tables in Catalog Explorer.

How to enable anomaly detection in Databricks

The setup is genuinely a few clicks. Open the schema in Catalog Explorer, go to the Details tab, click Enable, confirm that Anomaly detection is toggled on in the Data Quality Monitoring dialog, and save. An initial scan runs on its own. You need a Unity Catalog enabled workspace with serverless compute available, and MANAGE SCHEMA or MANAGE CATALOG privileges on the schema to turn it on. Viewing the health indicators only needs SELECT or BROWSE.

Cost is worth understanding before you enable it across a catalog. Anomaly detection runs on serverless compute and is billed as serverless DBUs under the DATA_QUALITY_MONITORING billing product, so it appears as a separate line you can track. The system table that stores results does not incur storage charges. It also does not modify your tables or add overhead to your existing jobs, which is the right design: monitoring that slows down the pipelines it watches gets turned off within a quarter.

What is Databricks Lakehouse Monitoring?

Lakehouse Monitoring is the more configurable layer. Instead of a schema-wide toggle, you attach a monitor to a specific table and pick a profile type. A snapshot monitor computes statistics over the whole table each run. A time series monitor computes them per time window using a timestamp column, which is what you want for fact tables that grow. An inference monitor is built for model outputs, tracking prediction drift and, once labels arrive, model quality.

Each monitor produces two Delta tables, a profile metrics table and a drift metrics table, plus an auto-generated dashboard. The profile table holds per-column statistics (counts, nulls, distinct values, quantiles, min and max) for each window and slice. The drift table compares a window against a baseline or against the previous window using statistical distance measures. Because these are ordinary Delta tables in Unity Catalog, you can query them, alert on them with Databricks SQL alerts, or pull them into whatever dashboard your team already opens.

The tradeoff against anomaly detection is coverage versus depth. Lakehouse Monitoring tells you far more about a table, including distribution changes on specific columns, but somebody has to create and maintain a monitor per table and choose the slicing expressions and baselines. That is the same maintenance curve as hand-written tests, and it bends the same way: coverage stalls at the tables an engineer had time for.

Databricks anomaly detection vs Lakehouse Monitoring vs Delta Live Tables expectations

There is a third native option people forget, so it helps to line all three up.

Delta Live Tables expectations are assertions you declare inside a pipeline, for example that a column must not be null or a value must fall in a range. You choose whether a violation warns, drops the row, or fails the pipeline. They run at write time and stop bad data from landing, which no monitoring product can do after the fact. The limit is that they only cover what you asserted, and only inside DLT pipelines.

Anomaly detection runs after the fact, needs no configuration, and covers every table in an enabled schema, but it only answers two and a half questions: is this table late, is it short, and (in beta) did null rates move.

Lakehouse Monitoring also runs after the fact, covers far more statistical ground per table, and requires per-table setup.

The sensible pattern is to use all three: expectations for the invariants you know and can enforce at write time, anomaly detection as a cheap safety net across every schema, and Lakehouse Monitoring on the tables that feed revenue reporting or a production model. What you are left with after that is still a real gap, and it is worth naming.

What Databricks native monitoring does not cover

Four things, in the order they tend to bite.

Schema drift. Neither anomaly detection nor Lakehouse Monitoring alerts you when an upstream source adds, drops, renames, or retypes a column. A dropped column often does not change the row count or the commit schedule at all, so the table looks perfectly healthy while a downstream model silently produces nulls. This is one of the most common causes of a wrong dashboard, and it is invisible to a freshness and volume check.

Coverage outside Databricks. Most data teams do not live in one platform. If your ingestion lands in Snowflake or a Postgres replica before it reaches the lakehouse, or your BI layer reads from somewhere else, native Databricks monitoring stops at the workspace boundary. It also does not cover views or foreign tables, and freshness based on an event-time column rather than commit time is not part of the current feature.

Lineage-based impact. Unity Catalog does capture lineage, and that is genuinely useful, but the monitoring output and the lineage graph are not joined into an incident. When a table is flagged stale you still have to go look up what depends on it and decide who to tell. The data lineage diagram and the alert live in different places, so the blast radius is a manual lookup at exactly the moment you have least time for one.

Alert routing and noise control. Health indicators in Catalog Explorer are a pull mechanism: they are correct and nobody is looking at them at 2am. You can wire Databricks SQL alerts onto the metric tables, but grouping related failures into a single incident, escalating by severity, and routing to the right on-call rotation is work you assemble yourself. It is the same discipline you would apply to any production dependency, in the same way you would keep an uptime check on the source API you ingest from rather than finding out from a stakeholder.

How to fill the gap

If your stack is entirely Databricks and you mainly worry about late and short tables, turn on anomaly detection, add DLT expectations to your critical pipelines, and stop there. It is free of configuration effort and it catches a real class of failure.

If you are running Databricks alongside another warehouse, or schema drift and wrong numbers are the failures that actually hurt you, you want monitoring that spans the whole stack and covers all five pillars: freshness, volume, schema, distribution, and lineage. That is the category the data quality tools market exists to serve, and an observability platform is the part of it that watches production tables continuously rather than asserting rules at write time.

Dataobservability connects to Databricks with a read-only role in about 15 minutes, profiles every table it can see, and generates monitors automatically instead of asking you to define them per table. It watches all five pillars, including the schema changes native monitoring skips, builds column-level lineage so every alert arrives with the downstream models and dashboards it affects, and groups related failures into one incident routed to Slack or PagerDuty. It reads warehouse metadata before it touches rows, which keeps compute small. Pricing is published and starts at 99 dollars a month, with a 14-day trial and no credit card, which is worth noting because most platforms in this category (Monte Carlo, Bigeye, Anomalo, Sifflet, Acceldata) will not quote you a number without a sales call.

Frequently asked questions

What is Databricks anomaly detection?

Databricks anomaly detection is a Unity Catalog feature that monitors every table in an enabled schema for freshness and completeness. It analyzes commit history to predict when a table should next update and flags it stale if the commit is late, and it predicts an expected row-count range over 24 hours and flags the table incomplete if actual rows fall below the lower bound. A percent-null check is available in beta. Results land in a system table.

How do I enable data quality monitoring in Databricks?

Open the schema in Catalog Explorer, go to the Details tab, click Enable, make sure Anomaly detection is toggled on in the Data Quality Monitoring dialog, and save. You need a Unity Catalog enabled workspace with serverless compute and MANAGE SCHEMA or MANAGE CATALOG privileges. An initial scan runs automatically and health indicators appear next to your tables shortly after.

How much does Databricks anomaly detection cost?

It runs on serverless compute and is billed as serverless DBUs under the DATA_QUALITY_MONITORING billing product, so the spend is visible as its own line rather than buried in your general compute. The system table that stores results does not incur storage charges. Intelligent scanning prioritizes higher-impact tables and skips low-impact ones, which is what keeps the bill reasonable when you enable a large schema.

What is the difference between anomaly detection and Lakehouse Monitoring?

Anomaly detection is schema-wide and needs no configuration, but only covers freshness, completeness, and beta null rates. Lakehouse Monitoring is per table and requires you to create a monitor and choose a profile type, but computes far richer profile and drift statistics per column and window, writes them to Delta metric tables, and generates a dashboard. Use anomaly detection as broad coverage and Lakehouse Monitoring on your most important tables.

Does Databricks detect schema changes?

Not as part of data quality monitoring. Anomaly detection covers freshness and completeness, and Lakehouse Monitoring computes column statistics on monitors you define, but neither alerts you when an upstream source adds, drops, renames, or retypes a column. A dropped column frequently leaves row counts and commit timing unchanged, so the table reads as healthy while downstream models quietly fill with nulls. Catching that needs schema monitoring on top.

Do I need a data observability tool if I use Databricks?

Not if your stack is entirely Databricks and late or short tables are your main failure mode; anomaly detection plus Delta Live Tables expectations covers that well. You do need one if you run more than one warehouse, if schema drift and silently wrong numbers are the failures that hurt, or if you want alerts that arrive with column-level lineage and downstream impact attached rather than a health indicator nobody is watching overnight.