Data Warehouse Incident Detection: The 4 Signals That Work

Data warehouse incident detection means noticing that your data is wrong before the people who use it do. The uncomfortable part is that most teams monitor the wrong layer. Orchestration monitoring tells you whether a job ran, and the majority of real data incidents happen with every job green: the pipeline succeeded, dbt passed, and the numbers are still wrong. Detecting incidents reliably means watching the data itself across four signals, freshness, volume, schema, and distribution, and routing what fires to whoever owns the dataset.

Why a green pipeline misses most data incidents

Ask a data team how they would know if something broke last night and the honest answer is usually some combination of Airflow data quality checks, dbt test failures, and a message from someone in finance. The first two only cover failures the pipeline can see. If an upstream API starts returning an empty array instead of an error, ingestion succeeds and loads zero rows. If a source system changes a currency field from dollars to cents, every row loads perfectly. If a partition arrives late, the job that reads it runs on time against incomplete data and reports success.

None of those trip a job failure. All of them are incidents, and the third one is the category that does the most reputational damage, because the number on the dashboard is plausible. Nobody questions a revenue figure that is 8 percent low. They question it three weeks later, in a meeting, and then the conversation is no longer about a pipeline bug.

This is the practical case for detecting at the data layer rather than the orchestration layer. Job monitoring answers whether the machinery ran. Incident detection answers whether the output is sane.

The four signals that catch warehouse incidents

Almost every data incident shows up as a change in one of four properties of a table. They are worth separating because they catch genuinely different failures and they cost very different amounts to implement.

The first three are cheap, and every team should have them before anything more sophisticated. They answer from table metadata rather than from scanning contents, so coverage across a thousand tables costs close to nothing in warehouse compute. Distribution monitoring is the expensive one and it is also the one that catches the silent incidents, which is the reason it exists.

How do you detect a data incident before stakeholders do?

Measure your current detection path honestly first. For the last five incidents, write down who noticed and how. If the answer is a person outside the data team more than once, your mean time to detection is measured in days and no amount of tuning will fix it, because there is no detector to tune.

From there the sequence that works is boring and effective. Start with freshness and volume across every table, not just the important ones, because the table that breaks is reliably the one nobody classified as important. Add schema change alerts on tables with downstream consumers. Only then add distribution monitoring, and add it first to the tables that feed anything a customer or an executive sees. Teams that start with distribution monitoring across the whole warehouse generate so many alerts in week one that the channel gets muted, and a muted channel is worse than no channel because it produces the feeling of coverage without the fact of it.

What is a good mean time to detection for data incidents?

The useful target is shorter than your reporting cycle. If leadership looks at a dashboard every Monday, detection has to happen inside a week to prevent a wrong decision, and inside a day to prevent an awkward conversation. In practice, freshness and volume checks running hourly get most teams to detection within an hour of the break, which is enough for the on call engineer to fix it before the business day starts. Chasing minutes is rarely worth it for analytics data, and it is the wrong optimization while any incidents are still being reported by stakeholders.

Track two numbers rather than one. Time to detection tells you whether monitoring works. Time to resolution tells you whether the alert carried enough context to act on, and it is the number that exposes alerts that fire into a channel with no owner attached.

Detection is only half of it: routing and impact

An alert that says orders_daily row count outside expected range at 3am is a detection success and an operational failure if nobody knows who owns orders_daily or what reads it. Two things turn a detection into a resolved incident.

The first is ownership routing. Alerts should go to the team that owns the dataset, in the channel they actually watch, which for most teams means Slack for the working day and PagerDuty for anything that cannot wait. A single shared alerts channel is where monitoring goes to die.

The second is downstream impact. The question an engineer asks immediately after seeing an alert is what breaks because of this. Answering it by reading SQL takes an afternoon. Answering it from column level lineage takes seconds, and the difference decides whether you can tell stakeholders which reports to distrust before they open them. This is also what turns a stream of alerts into a small number of owned incidents: a single upstream break that fires eleven monitors is one incident with eleven symptoms, and grouping it that way is the job of data incident management rather than of the detector.

Worth noting that not every warehouse alarm is a data quality problem. A runaway query that scans a partition it should have pruned is an incident too, just a financial one, and it belongs with the tooling teams use for tracking unexpected cloud spend rather than in the data quality channel. Mixing the two is a common way to make both sets of alerts easier to ignore.

Doing this on your warehouse

Every major warehouse exposes the metadata these signals need, and none of them ships the monitoring system on top.

On BigQuery, table metadata and job history give you freshness and volume without scanning contents, and the native options for finding outliers are BQML functions and rule based data quality scans in Knowledge Catalog. Both find things; neither schedules, deduplicates, routes, or escalates, which is the gap covered in more depth on BigQuery anomaly detection.

On Snowflake, Data Metric Functions cover row count, null count, duplicate count, and freshness, attached per object on a schedule, and they require Enterprise Edition. On Databricks, Unity Catalog powers data quality monitoring with results landing in a system table, billed as serverless compute. On Redshift there is less native ground: SYS_QUERY_HISTORY gives you what ran and SVV_TABLE_INFO gives you scan free row estimates, and the rest is yours to build.

The common shape is that each warehouse gives you the raw signal and leaves the system around it undone. Whether you build that system or buy it comes down to how many tables you have and whether you have an engineer to keep it alive, which is the calculation laid out in build versus buy for data observability.

Frequently asked questions

What is data warehouse incident detection?

Data warehouse incident detection is the practice of automatically identifying when data in a warehouse has broken, using signals from the data itself rather than from the pipeline that loaded it. The four standard signals are freshness, volume, schema, and distribution. It is distinct from pipeline monitoring, which reports whether jobs ran, because most data incidents occur while every job reports success.

How is data incident detection different from pipeline monitoring?

Pipeline monitoring watches the orchestration layer and answers whether a job completed. Incident detection watches the data layer and answers whether the output is correct. The distinction matters because failures like an empty upstream API response, a unit change from dollars to cents, or a late partition all produce successful job runs and wrong data. You need both, but only one of them catches silent corruption.

Which signal catches the most data incidents?

Freshness and volume catch the largest share of incidents for the least effort, because outages and partial loads are the most common failure modes and both answer from table metadata rather than table scans. Distribution monitoring catches fewer incidents but catches the most damaging ones, the silent cases where data loads on time, in the right quantity, with the wrong values.

How long should it take to detect a data incident?

Faster than your reporting cycle. Hourly freshness and volume checks put most teams inside an hour of the break, which is generally enough to fix things before the business day starts. The more important measure early on is what share of incidents are found by monitoring rather than reported by a stakeholder; until that number is high, tuning detection latency is premature.

Can dbt tests handle incident detection?

Partly, and they are worth having. dbt tests are assertions you write, so they cover the conditions you thought of, on the models inside your dbt project, at the moment the run executes. They do not cover tables outside the project, they do not learn what normal looks like per table, and they do not run between dbt invocations. They work well alongside baseline monitoring and poorly as a replacement for it.

Do I need distribution monitoring or is freshness enough?

Freshness is enough if your main risk is pipelines stopping. It is not enough if wrong numbers reaching a dashboard would be worse than missing numbers, because a table can be perfectly fresh, perfectly sized, and completely wrong. A useful test: if your last serious incident involved data that arrived on time and looked normal, freshness monitoring would not have caught it.