Data Observability Architecture: The Layers and a Reference Design

A data observability architecture is the set of components that continuously collect metadata about your data platform, detect when that metadata deviates from normal, explain the blast radius of a problem, and route it to a human. It has five layers: collection (reads metadata and query history from your warehouse and orchestrator), detection (runs rules and statistical models against that metadata), lineage (maps table and column dependencies), incidents (deduplicates and routes alerts), and the interface (where humans see the graph and the history). It sits beside your pipeline as an observability plane, not inside it, so it can go down without taking your data with it.

That last point separates a real architecture from a pile of tests: if your monitoring lives inside your DAG, a broken monitor breaks the pipeline, and a broken pipeline means no monitoring. Last updated July 2026.

What is data observability architecture?

Data observability architecture is the layered design that lets you know the state of every table in your platform without querying every table. It collects metadata (row counts, last modified timestamps, schemas, query logs), applies detection logic to it, resolves lineage to determine impact, and routes incidents to owners. It covers five pillars: freshness, volume, schema, distribution, and lineage.

The word "architecture" matters because the naive version does not scale. Write a freshness check for 4,000 tables and you have a scheduling problem, a compute cost problem, and an alert fatigue problem.

What are the layers of data observability?

The architecture diagram, described

A data observability architecture diagram, described in words. The flow:

1. Sources. Production Postgres, Salesforce, Stripe, event streams. Where most incidents originate.
2. Ingestion. Fivetran, Airbyte, CDC, custom loaders. Emits run status, rows loaded, sync durations.
3. Warehouse. Snowflake, BigQuery, Databricks, or Redshift. Its metadata catalog already knows every table's row count, byte size, last altered time, and every query run against it.
4. Transformation. dbt under Airflow or Dagster. Emits a manifest (the DAG and columns) and run results.
5. Serving and BI. Looker, Tableau, Power BI, reverse ETL. Where a bad number becomes a bad decision.
6. The observability plane. Sits to the side of all five, read-only. It polls warehouse metadata, ingests the dbt manifest after each run, parses query logs to derive column-level data lineage, and pushes alerts to Slack and PagerDuty.

Draw it as a horizontal pipeline with a bar underneath spanning the full width, arrows up from every stage into the bar, one arrow out into Slack.

Metadata-first vs query-based collection

Query-based collection runs SQL against the data itself: SELECT COUNT(*), SELECT MAX(updated_at), null rate scans. It is what dbt tests and Great Expectations do. Accurate, flexible, expensive. A COUNT(*) on a large Snowflake table spins up a warehouse and scans partitions. Do that every 15 minutes across 2,000 tables and you have built a cost center. Those scans land on your Snowflake bill, which is usually where whoever watches cloud and warehouse spend notices them first, weeks before anyone connects the monitoring rollout to the line item.

Metadata-first collection asks the warehouse what it already knows. Snowflake's ACCOUNT_USAGE.TABLES gives you ROW_COUNT, BYTES, and LAST_ALTERED for every table, no scan of the data. BigQuery and Databricks expose the same through INFORMATION_SCHEMA and Delta history.

So go metadata-first by default, query-based only where you need it. Three of the five pillars come essentially free that way. Distribution checks do have to touch data, so run them on a sample, on the columns that matter, on a slower cadence. Our collector works this way over a read-only role, which is how data quality monitoring covers thousands of tables cheaply.

Push vs pull, agent vs agentless

Pull means the plane connects out and asks. Push means your pipeline emits events to it (dbt artifacts after a run, an Airflow callback on failure). You want both: pull covers tables nobody instrumented, push carries context only the pipeline has.

Agentless means the platform holds a read-only credential and queries over the network. It is a 15-minute setup and what most teams should do. Agent-based means a collector inside your VPC that ships only metadata out. Use one when your warehouse has no public endpoint or compliance forbids external credentials, knowing it is one more thing to deploy and patch.

How does data observability get lineage?

Three sources, in decreasing order of coverage and increasing order of precision. Query logs give you edges from the SQL the warehouse actually ran, the dbt manifest gives you the transformation DAG as defined, and BI APIs extend the graph to dashboards.

• Query log parsing. Snowflake's QUERY_HISTORY and ACCESS_HISTORY, BigQuery's audit logs. Parse the SQL, resolve the ASTs, and you get lineage for everything, including the ad hoc CREATE TABLE AS that became load bearing.
• The dbt manifest. Precise, includes columns, tests, and owners, and free. But it only covers what dbt builds, so if half your warehouse arrives via Fivetran and stored procedures, it sees none of it.
• BI metadata. Looker and Tableau APIs. These let an alert say "this breaks the CFO's revenue dashboard" instead of "table fct_orders_v2 is stale."

Column-level lineage beats table-level because it collapses the blast radius. Table-level says 47 models depend on fct_orders. Column-level says only 3 touch the column that changed: a fifteen minute fix instead of a war room.

Where alerting and routing sit

Alerting is its own layer, not a feature bolted onto detection. The hard problem is noticing 200 failures caused by one upstream break and sending one message. That takes three things.

• Lineage-aware grouping. If 40 tables went stale and all descend from one failed load, that is one incident with 40 affected assets, not 40 pages.
• Ownership resolution. A mapping from asset to team, ideally from dbt owner meta, so routing needs no human.
• Severity that reflects consumption, not table size. A stale staging table nobody reads is a ticket. A stale table feeding a board dashboard is a page. Route the first to Slack, the second to PagerDuty.

Should you build or buy data observability?

Build if observability is a core competency you intend to staff permanently, or if your platform is so unusual no vendor connects to it. Buy otherwise. The honest math: a DIY stack of dbt tests, Airflow sensors, and Great Expectations gets you maybe 60 percent of the value for a permanent fraction of an engineer, and never gets you lineage or learned baselines.

• dbt tests cover assertions you thought of, on models dbt builds. They miss raw ingestion tables, never learn baselines, and every test is a query you pay for on every run.
• Airflow sensors tell you a task failed. They do not tell you a task succeeded while writing 40 percent of the usual rows, the more common and more damaging failure.
• Great Expectations is good at declarative validation, and query-based by design, so cost grows with tables times columns times frequency.
• The missing pieces are the expensive ones: column-level lineage from parsed query logs, learned baselines, incident deduplication, and a UI a non-engineer can read. Quarters of work that do not differentiate you.

The real comparison is a recurring slice of a senior engineer against a subscription. Our pricing starts at $99 a month, a rounding error next to the loaded cost of the engineer who would otherwise write and maintain homegrown freshness checks. Enterprise platforms are typically sold contact-sales, so know what data observability tools actually cost before you sit through six demos, and give the wider data quality tools landscape a look.

Reference architecture: Snowflake, dbt, Airflow, BI

1. Grant a read-only role. SELECT on SNOWFLAKE.ACCOUNT_USAGE and the schemas you cover. No write grants.
2. Poll metadata every 5 to 15 minutes. ACCOUNT_USAGE.TABLES returns row counts, sizes, and last altered times for the whole account in one query.
3. Ingest dbt artifacts after every run. manifest.json for the DAG, columns, and owners; run_results.json for test outcomes. Airflow pushes these on task completion.
4. Parse QUERY_HISTORY and ACCESS_HISTORY nightly to build column-level lineage across everything dbt does not build, and to attach BI consumption to tables.
5. Build baselines from the collected series. Seven to fourteen days is enough to model a table's normal cadence and volume with day-of-week seasonality, which is where anomaly detection beats thresholds that scream or stay silent.
6. Sample distribution checks selectively. Null rates and value ranges on the 50 to 200 columns that carry business meaning, not all 40,000.
7. Route by lineage and ownership. DAG failures into pipeline monitoring, table anomalies into the owning team's Slack channel, anything touching a revenue asset into PagerDuty.

Add data contracts at the producer boundary and you cover both halves: contracts stop the failures you can specify in advance, observability catches the ones nobody wrote down.

Start with the layer you are missing

Most teams have some detection (dbt tests), no collection layer, no lineage, no incident routing. That is why their alerts are noisy and their root cause analysis is a Slack thread. Fix collection first: everything else builds on it.

If you would rather see this on your own warehouse than in a diagram, our 14-day free trial connects read-only to Snowflake, BigQuery, Databricks, or Redshift and gives you freshness, volume, and schema coverage plus column-level lineage in about 15 minutes. If it surfaces nothing you did not already know in week one, it is not for you.