Integration
Databricks Data Observability for the Lakehouse
Monitor your Databricks lakehouse for freshness, volume, schema, and anomalies across Delta tables and Unity Catalog.
14-day trial, no credit card, read-only connection
Alerted #data-eng 0.8s ago.
Downstream impact · consumers at risk
In one paragraph
Databricks data observability means monitoring your Delta tables and lakehouse for freshness, volume, schema, and anomaly issues, plus lineage across Unity Catalog. Dataobservability connects with read-only access, uses table and catalog metadata to keep cluster cost low, and alerts the data team the moment a table breaks.
Side by side
Databricks monitoring options compared
Swipe to see all columns →
| Option | What it covers | What it misses | Cost model |
|---|---|---|---|
| Anomaly detection (Unity Catalog) | Freshness and completeness on every table in an enabled schema, plus percent-null in beta | Schema drift, distribution shifts on arbitrary columns, cross-warehouse lineage | Serverless DBUs, billed under DATA_QUALITY_MONITORING |
| Lakehouse Monitoring (Data Quality Monitoring) | Profile and drift metrics on tables you configure, written to metric tables with a dashboard | Per-table setup effort, no automatic coverage of a whole catalog | Serverless compute per monitor refresh |
| DLT expectations | Row-level rules inside declarative pipelines, with drop or fail actions | Only tables built by DLT, only rules you wrote | Included with pipeline compute |
| Databricks Labs Overwatch | Cluster, job, and cost telemetry for platform operations | Not a data quality tool, and moving toward deprecation as system tables take over | Open source, you run it |
| Dataobservability | All five pillars across Delta tables plus lineage and grouped incidents | Nothing Databricks-native that you would also want twice | Public pricing from $99/mo |
Summarized in good faith from public documentation, August 2026. Verify current behavior and terms with the vendor.
Why it fits
Lakehouse teams on Databricks who need observability without heavy cluster spend.
Built for the lakehouse
Monitors cover Delta tables and read Unity Catalog metadata for lineage.
Low cluster footprint
Metadata-first checks avoid expensive full-table scans.
Alerts that group
Related anomalies collapse into one incident, routed to Slack or PagerDuty.
What Databricks gives you natively, and where it stops
Databricks has genuinely improved here. Anomaly detection in Unity Catalog watches freshness and completeness across every table in a schema without per-table thresholds, and writes results to the system.data_quality_monitoring.table_results system table. Lakehouse Monitoring, now presented as Data Quality Monitoring, computes profile and drift metrics on tables you explicitly configure and builds a dashboard from them. DLT expectations enforce row-level rules inside declarative pipelines. Between them you get two of the five pillars covered well: freshness and volume. What is still missing is schema drift detection on tables outside DLT, distribution monitoring on columns you did not pick in advance, and lineage that reaches past the lakehouse into the BI layer where the damage is actually noticed. That gap is the reason lakehouse teams still buy an observability platform after enabling everything Databricks ships.
Databricks observability without the cluster bill
The fastest way to make lakehouse monitoring expensive is to scan rows. A tool that runs SELECT COUNT and full profiling passes across a catalog of a few thousand Delta tables burns real compute every cycle, and the finance conversation that follows usually ends with monitoring being scaled back to a handful of tables. Dataobservability reads Unity Catalog metadata, Delta transaction history, and table statistics first, and only samples rows when a metric genuinely requires it. Commit history already tells you when a table last changed, and Delta statistics already carry row counts and null counts per file, so most freshness and volume monitoring needs no scan at all. That is what makes full-catalog coverage affordable instead of a pilot on twelve tables.
Overwatch, system tables, and what to use in 2026
Teams searching for Databricks observability often land on Overwatch, the Databricks Labs open-source project that collected cluster, job, and cost telemetry into Delta tables. It is worth knowing that Overwatch is platform observability, not data observability: it tells you what your clusters and jobs did, not whether the data they produced is correct. It is also being superseded, as Databricks moves that operational telemetry into Unity Catalog system tables, which are governed, hosted, and queryable without you running a collector. The practical 2026 answer is to read system tables for platform and cost questions, enable Unity Catalog anomaly detection for baseline freshness and completeness, and add a data observability platform for schema, distribution, lineage, and incident workflow.
What Unity Catalog lineage captures, and the specific cases where it goes blank
Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, with no configuration. That is genuinely valuable and it is one of the strongest native lineage stories of any warehouse. It is also bounded in ways that are documented but easy to miss until the graph disappoints you mid incident, so they are worth knowing in advance. Databricks states plainly that lineage data captured before September 1, 2024 is not available, so anything you are trying to reconstruct from earlier history simply is not there. Lineage is not preserved for renamed catalogs, schemas, tables, views, or columns, which means a routine refactor silently severs the graph at the point of the rename. Several execution paths produce no lineage at all: jobs submitted through the Jobs API runs submit request or the spark submit task type, Resilient Distributed Datasets, global temp views, and anything using Spark SQL dataset checkpointing. Column level lineage in particular requires both the source and the target to be referenced by table name rather than by path, so code that reads or writes a location string rather than a catalog object drops out. User defined functions are another blind spot, because a UDF obscures the mapping between source and target columns even when both sides are visible. Retention differs by surface too: lineage stays in Catalog Explorer indefinitely for everything captured after September 2024, while the system tables keep a rolling one year window. None of this makes Unity Catalog lineage a poor tool. It makes it a tool with a shape, and the practical response is to test your own awkward assets against it early, then decide whether the gaps matter enough to want a graph that is also parsed from query history.
Delta table freshness: why the last commit time is not the same as fresh data
The obvious way to monitor freshness on a Delta table is to read the latest commit timestamp from the table history and alert when it gets too old. It works until it quietly stops working, and the reason is that Delta records a new table version for operations that do not change the logical contents at all. An OPTIMIZE run that compacts small files writes a new version. So does VACUUM cleaning up expired files, and so does a metadata change like setting table properties. A table whose upstream job has been failing for two days can therefore look freshly written because a scheduled maintenance job touched it overnight, which is precisely the failure mode that lets a stale dashboard survive a whole reporting cycle unnoticed. The fix is to distinguish the operation type rather than trusting the timestamp: Delta history records what each version did, so a freshness check should care about the operations that actually wrote data and ignore the ones that only reorganized it. Streaming tables add a second wrinkle in the opposite direction. A structured streaming sink may commit constantly while producing almost no rows, so commit recency looks perfect while volume has collapsed. That is why freshness and volume have to be read together on a lakehouse rather than treated as independent checks, and why a monitor tuned to a batch table applies badly to a streaming one. The useful pattern is to learn each table's normal write cadence and normal row delta from its own history, including its weekly seasonality, and alert on departures from that learned shape rather than on a fixed interval somebody guessed at setup time.
How Dataobservability connects to Databricks
Connection is a read-only service principal with USE CATALOG and SELECT on the schemas you want watched, plus access to Unity Catalog metadata. There is no agent to install, nothing to run inside your workspace, and no change to your existing jobs or pipelines. Monitors are generated automatically from what it finds: every Delta table gets freshness and volume monitors tuned to its own history, schema snapshots are diffed on every check so a dropped or retyped column is caught immediately, and column-level lineage is built by parsing the SQL behind your views, notebooks, and dbt models. Alerts group by root cause and route to Slack or PagerDuty. Setup is a read-only connection, and pricing is published: 99 dollars a month to start, no sales call required.
Frequently asked questions
What is Databricks data observability?
Databricks data observability is continuous monitoring of the data in your lakehouse across freshness, volume, schema, distribution, and lineage, rather than monitoring of clusters and jobs. It answers whether the Delta tables your pipelines produced are correct and current, and which downstream models and dashboards are affected when they are not. Unity Catalog covers part of this natively; a dedicated platform covers the rest.
Does Databricks have built-in data quality monitoring?
Yes. Unity Catalog anomaly detection monitors freshness and completeness across every table in an enabled schema, and Lakehouse Monitoring, now branded Data Quality Monitoring, computes profile and drift metrics on tables you configure. Delta Live Tables expectations add row-level rules inside pipelines. The gaps are schema drift outside DLT, distribution monitoring on columns you did not pick, and lineage past the lakehouse.
What is Databricks Lakehouse Monitoring?
Lakehouse Monitoring is the Databricks feature that profiles a table on a schedule and tracks how its statistics drift over time. You create a monitor on a specific table, choose a profile type, and Databricks writes profile and drift metrics to generated metric tables and builds a dashboard. It reached general availability in August 2024 and is now presented as Data Quality Monitoring, powered by Unity Catalog.
Is Databricks Overwatch still the right way to monitor Databricks?
Overwatch is a Databricks Labs open-source project that collected cluster, job, and cost telemetry into Delta tables, and it is moving toward deprecation as that telemetry becomes available directly in Unity Catalog system tables. It was never a data quality tool. For platform and cost questions use system tables; for whether the data itself is correct, use anomaly detection plus a data observability platform.
How much does monitoring Databricks cost?
Native anomaly detection and Lakehouse Monitoring run on serverless compute and are billed as DBUs, with anomaly detection appearing under the DATA_QUALITY_MONITORING billing product so you can track it separately. A third-party platform adds a subscription. Most of the real cost difference comes from whether a tool scans rows or reads metadata: metadata-first checks keep the compute line small even across a few thousand Delta tables. Dataobservability publishes its pricing from 99 dollars a month.
Does data observability work with Unity Catalog?
Yes, and Unity Catalog makes it considerably better. Unity Catalog is where table metadata, commit history, and access grants live, so an observability platform can read freshness and volume signals without touching rows, and can resolve lineage across catalogs and schemas. Dataobservability connects through a read-only service principal with USE CATALOG and SELECT, and needs no agent inside your workspace.
Does Unity Catalog have column level lineage?
Yes. Unity Catalog captures lineage automatically for queries run on Databricks down to the column level, with no setup required. The documented limits matter: nothing captured before September 1, 2024, lineage is not preserved through renames of catalogs, schemas, tables, views, or columns, and RDDs, global temp views, checkpointed datasets, and user defined functions are not captured. Column lineage also requires both source and target to be referenced by table name rather than by path.
How long does Databricks keep lineage data?
It depends where you read it. In Catalog Explorer, lineage is retained indefinitely for everything captured after September 1, 2024, so the browsable graph does not expire. The lineage system tables keep a rolling one year window instead, which matters if you are querying them programmatically for audit evidence or trend analysis. Anything from before September 2024 was never captured and cannot be recovered.
Why did my Databricks lineage disappear after renaming a table?
Because Unity Catalog does not preserve lineage through renames. Renaming a catalog, schema, table, view, or column breaks the recorded relationships at that point, and the graph starts rebuilding from subsequent queries rather than carrying the history forward. This catches teams during refactors, when the graph is most needed. Lineage parsed independently from query history is not subject to the same reset, which is one reason to run both.
Can I detect anomalies in Databricks without writing thresholds?
Yes. Unity Catalog anomaly detection builds its own per-table model of commit timing and row counts, so you enable it on a schema rather than configuring each table. Dataobservability does the same across all five pillars, learning each Delta table's normal freshness, row count, null rate, and value distribution from history, then alerting when behavior leaves the learned band. Neither approach requires you to guess a number.
Live on day one
Connect your warehouse and watch monitors generate across every table. Transparent pricing, no credit card.