Integration
Airflow Monitoring Tied to Your Data Quality
Connect Airflow run signals to warehouse freshness and volume, so a late DAG turns into a data incident, not a silent stale table.
14-day trial, no credit card, read-only connection
Alerted #data-eng 0.8s ago.
Downstream impact · consumers at risk
In one paragraph
Airflow monitoring in a data observability context means correlating your DAG and task outcomes with the freshness and volume of the tables they produce. Dataobservability reads orchestration signals alongside warehouse metadata, so when a job fails or runs late you see exactly which downstream tables and dashboards are at risk.
Side by side
Airflow data quality and monitoring options compared
Swipe to see all columns →
| Option | What it watches | Setup | Blind spot |
|---|---|---|---|
| Airflow task state and retries | Whether each task ran, failed, retried, or timed out | Native to every DAG | A task can succeed and still land wrong or zero rows |
| SQLColumnCheckOperator and SQLTableCheckOperator | Column and table assertions you write: null rate, uniqueness, row count, min and max | Add the operator to the DAG with a column_mapping and a connection | Only the columns and tables you wrote checks for, on the runs that fire |
| GreatExpectationsOperator | Expectation suites run as a DAG task | Author suites, install the provider, wire the task | You maintain the suites, and it only runs inside the DAG |
| OpenLineage from Airflow | Run-level lineage events, plus data quality facets from the SQL check and GX operators | Enable the OpenLineage provider, send events to a backend | Lineage and metadata, not baselines, alerting, or coverage of untested tables |
| SLA misses and deferrable sensors | Whether a task finished inside its expected window | Configure per task or DAG | Says a job was late, not whether the data it produced is correct |
| Data observability platform (Dataobservability) | Freshness, volume, schema, and distribution on every table, with learned baselines and column level lineage | Read only warehouse connection, monitors generate automatically | Published pricing from $99 a month |
Summarized in good faith from public documentation, August 2026. Verify current behavior and terms with the vendor.
Why it fits
Teams orchestrating with Airflow who want job health joined to data health.
DAGs linked to tables
A late or failed task is connected to the freshness of the tables it loads.
Beyond job success
A job can succeed and still deliver bad data. Volume and anomaly monitors catch that.
Impact-first alerts
Lineage shows what breaks downstream when a pipeline misses its window.
Airflow monitoring versus data observability
Airflow is an orchestrator, and its native monitoring answers the orchestrator question: did the task run, did it fail, did it retry, did it finish inside its window. That is genuinely useful and you should watch it, because a DAG that failed at 2am is a real problem. But a green DAG is not the same as correct data, and the gap between those two is where most data incidents live. A task that runs a COPY or an INSERT reports success the moment the SQL returns without an error, whether it loaded forty thousand rows or zero. An upstream API that started returning a new category value does not fail any task; it just quietly reshapes a column three transformations later. A source that paused overnight means the DAG builds cleanly on stale data and every task is green. Orchestration monitoring watches the jobs. Data observability watches the tables the jobs produce: did this table update on schedule, did it receive its normal volume, did its schema change, did the important columns hold their usual shape. You want both, because a late DAG and a silently empty table are different failures that need different signals.
Data quality checks inside Airflow, and where they stop
Airflow does have first party data quality building blocks, and for a handful of critical tables they are worth using. The Common SQL provider ships SQLColumnCheckOperator and SQLTableCheckOperator: you give them a connection, a table, and a column_mapping describing the assertions, and they fail the task when a null rate, uniqueness, row count, or min and max bound is violated. There is also the GreatExpectationsOperator for running expectation suites as a task, and the plain SQLCheckOperator for arbitrary boolean SQL. These are real, in pipeline enforcement and they cost nothing beyond the compute of the checks. What they do not do is scale to coverage. Every check is a rule you wrote by hand, so the table a new pipeline created last month has no operator watching it and therefore reports no problems, which looks exactly like being healthy. The thresholds are static unless you maintain them, so a row count bound that was right in the spring fires constantly or never fires by the autumn. And they only run when the DAG runs, so a failure between scheduled intervals, or on a table no DAG touches, is invisible. In-DAG checks are the right tool for the few tables where you know precisely what correct looks like. They are the wrong tool for keeping an eye on a warehouse where new tables appear every sprint.
What Airflow 3 and OpenLineage changed
Airflow 3.0, which went generally available in 2025, moved the platform toward being data aware rather than purely task aware. Assets, the renamed and expanded version of Datasets, let a DAG declare the data objects it produces and consumes, so downstream DAGs can trigger on the asset being updated instead of a fixed schedule, and the UI can show the lineage between them. Airflow also emits OpenLineage events natively, and the SQL check operators and the GreatExpectationsOperator attach data quality facets to those events, so a lineage backend can record not just that a task ran but what it asserted and whether it passed. This is a meaningful step and it makes Airflow a much better citizen in a wider observability stack. It is still lineage and metadata plumbing rather than a monitoring product: OpenLineage tells you how tables connect and what happened on a run, but it does not learn each table's normal freshness and volume, it does not raise an incident when a table that always loads at 6am has not loaded by 9, and it does not deduplicate a hundred alerts down to the one broken upstream. Those are the parts a data observability platform adds on top, and the cleanest architecture is to let Airflow orchestrate and emit lineage while a monitoring layer reads the warehouse and owns the baselines and alerting.
Joining pipeline health to data health without building it yourself
The setup teams actually want is one where a failed or late Airflow job and a stale or wrong table are the same incident, routed to the same owner, with the downstream blast radius attached. Building that from parts means more than turning on a few operators. You need somewhere durable to store run history and table metrics, baselines that adapt per table instead of fixed thresholds, deduplication so one broken source does not page ten times, freshness derived from when a table actually last loaded rather than whether a task returned zero, and a lineage graph that answers who is affected the moment something breaks. Each piece is buildable and together they are usually a couple of engineer quarters plus ongoing maintenance, which is why the common pattern is to keep Airflow for orchestration and connect a monitoring layer that already does the rest. Dataobservability reads warehouse metadata with a read only connection, generates freshness, volume, schema, and distribution monitors across every table without you authoring rules, learns each table's normal behavior instead of asking for thresholds, maps the downstream impact with column level lineage, and publishes its pricing from 99 dollars a month, so a late DAG and the stale table it left behind show up as one alert with the affected dashboards named.
Frequently asked questions
What is Airflow monitoring?
Airflow monitoring, in a data context, means watching two different things: the health of your DAGs and tasks (did they run, fail, retry, or miss their window) and the health of the data those tasks produce (did the tables update on time, in the right volume, with the expected schema and values). Airflow covers the first natively. The second needs data quality checks or a data observability layer, because a task can succeed while the data it wrote is wrong.
Can Airflow do data quality checks?
Yes. The Common SQL provider includes SQLColumnCheckOperator and SQLTableCheckOperator for column and table assertions such as null rate, uniqueness, row counts, and value bounds, plus a general SQLCheckOperator and a GreatExpectationsOperator for expectation suites. They fail the task when an assertion breaks. The limit is coverage: they only check the tables and columns you wrote rules for, on the runs that fire, with thresholds you maintain by hand.
What is the difference between Airflow monitoring and data observability?
Airflow monitoring watches the orchestration: task state, retries, run duration, and SLA misses. Data observability watches the tables the orchestration produces: freshness, volume, schema, and distribution, measured against each table's learned baseline. A COPY that loads zero rows is a perfect result to Airflow and a serious incident to data observability, which is why teams run both rather than choosing one.
Does Airflow 3 support data lineage?
Yes. Airflow 3.0 emits OpenLineage events natively, and its Assets feature lets DAGs declare the data objects they produce and consume so downstream DAGs can trigger on updates and the UI can show the connections. The SQL check and Great Expectations operators attach data quality facets to those lineage events. This is lineage and metadata rather than monitoring: it records how tables connect and what happened, not whether a table is behaving normally.
Can a Airflow DAG succeed but still produce bad data?
Constantly, and it is the core reason to add data observability. A task succeeds when its SQL returns without an error, regardless of whether it loaded the expected rows. A COPY from an empty S3 prefix, an upstream API that changed a field, or a source that paused overnight all leave the DAG green while the table is empty, reshaped, or stale. Monitoring the data itself against its normal behavior is what catches these.
Should I use Airflow SQL check operators or a data observability tool?
Use both, for different jobs. SQL check operators are the right tool for a small set of critical tables where you know exactly what correct looks like and want the pipeline to stop on violation. A data observability tool is the right tool for coverage across a whole warehouse where new tables appear regularly, because it generates monitors automatically and learns baselines rather than asking you to write and maintain every rule.
How does Dataobservability work with Airflow?
It connects to your warehouse with a read only account and monitors the tables your Airflow DAGs produce, so a job that fails, runs late, or succeeds while landing wrong data surfaces as a data incident with the affected downstream dashboards attached. Airflow keeps orchestrating and can emit OpenLineage events; Dataobservability owns the freshness, volume, schema, and distribution baselines and the alerting, so you do not build that layer yourself.
Live on day one
Connect your warehouse and watch monitors generate across every table. Transparent pricing, no credit card.