Data Quality for AI: Monitoring the Pipelines Behind RAG and Agents

An AI system is only as trustworthy as the data it reads at inference time, and that data is usually the warehouse and document store the data team already owns. When a RAG app confidently cites a policy that was superseded in April, the model did not hallucinate: a sync job died and nobody noticed. A model cannot tell that the rows it retrieved are stale, duplicated, half-parsed, or missing. It answers anyway, fluently, and that fluency is what makes bad data more dangerous in an AI pipeline than in a dashboard.

Classic BI monitoring was built for tables humans look at. A stale table produces a chart someone eventually squints at. A stale embedding index produces a confident wrong answer a customer acts on. (Last updated July 2026.)

Why is data quality important for AI?

Because the model cannot detect bad input. Retrieval returns the top k chunks by vector similarity whether or not they are three months old, duplicated six times, or truncated mid-sentence by a parser regression. The model then generates a fluent answer grounded in that garbage. Garbage in, garbage out is not a slogan here, it is the literal control flow.

The second reason matters more to a head of data: attribution. When an AI feature is wrong, the instinct is to blame the model, so the ML team spends two weeks tuning prompts while the real cause (a source that stopped syncing on June 3) sits unfixed.

The failure modes classic monitoring misses

The five pillars (freshness, volume, schema, distribution, lineage) still apply. They just have to be measured at stages that did not exist in a BI stack:

• Stale embedding index. Source rows were updated. The vectors were not re-embedded. The warehouse table looks fresh, every dbt test passes, and the index answers from last quarter. No table-level check catches it: the broken artifact lives outside the warehouse.
• Silent chunking or parsing regressions. Someone bumps a PDF parser version and tables come out as a wall of concatenated digits. Document count is unchanged, ingestion succeeded, retrieval quality collapses for one document type. Row counts never show this. Mean chunk length does.
• A source that quietly stops syncing. The Confluence connector's token expired. It fails open, returns zero new documents, logs a clean success. Three months later the model still answers from the old policy, because the old policy is the only one in the index.
• Schema drift in a feature table. An upstream column changes type or starts arriving null. The training job coerces it, the serving path does not, and you get training/serving skew that shows up as a slow accuracy decay nobody can date.
• Duplicate documents inflating retrieval. The same policy PDF is ingested from three shares. Now all five retrieved chunks are one document, the context window is full of a single source, and the model looks confidently narrow.
• PII in the corpus. A support export with raw customer emails lands in the retrieval corpus. Nothing breaks, everything works, and you have a compliance incident a model will read aloud to whoever asks.
• Corpus distribution shift. A new source doubles the marketing copy in a corpus that was mostly technical docs, and retrieval starts pulling brochure language into engineering answers. Almost nobody monitors corpus composition.

Does bad data cause AI hallucinations?

Often, yes, and it is the tractable half of the problem. Models confabulate, but far more production hallucinations are the model faithfully summarizing retrieved context that is stale, duplicated, mis-parsed, or irrelevant. If the right chunk was never in the index, the model has no way to be right. That is a pipeline bug wearing a model bug's costume.

The test: take a failing answer and query the index directly for the ground-truth passage. If it is missing, or the version there is old, stop debugging the model. If the correct chunk was retrieved and ranked first and the answer is still wrong, you have a real model or prompt problem. That check routes most incidents in ten minutes.

Freshness SLAs for retrieval corpora and embeddings

Table freshness is solved. Embedding freshness is not, because it is a lag between two systems rather than a property of one. The metric you want is re-embedding lag: the time between a record's last update in the source of truth and the last time its vector was written to the index.

Keep source_updated_at and embedded_at in a manifest table keyed by document ID that your indexer writes back to. The monitor is then plain SQL:

select
  count(*) as stale_docs,
  max(datediff('minute', s.source_updated_at, i.embedded_at)) as max_lag_minutes
from source_documents s
join embedding_index_manifest i using (doc_id)
where s.source_updated_at > i.embedded_at

Set an SLA like you would for any table. A product docs corpus might be fine at 24 hours. A pricing corpus an agent quotes to customers should be under an hour, and stale_docs > 0 past that window should alert. Also watch documents never embedded at all, and vectors whose source row was deleted (orphans that keep getting retrieved forever).

Feature stores, training data, and skew

Training/serving skew is a data problem, not a modelling problem: the offline feature computed in a dbt model and the online feature computed in a service drift apart because two teams implemented the same logic twice. Sample serving-time vectors and compare their distributions to the training set.

Feature drift is distribution anomaly detection under another name: mean, null rate, and cardinality per feature against a trailing baseline. Run it in your data platform, not an ML-specific tool, because the next question after a drift alert is always "what changed upstream", and answering it needs column-level lineage back to the source. Drift detection alone says you have a problem. With lineage, it says which job caused it. Training corpora deserve the same checks.

What is data observability for AI?

It is applying the five pillars to every stage of an AI pipeline, not just the warehouse tables at the start of it. That means treating the chunking step, the embedding job, and the index itself as first-class data assets, monitored with the same alerting and sitting in the same lineage graph as your dbt models.

The market has landed here too. Monte Carlo now markets an autonomous, agent-facing observability platform, and Bigeye an enterprise AI trust platform. When both category leaders reframe around AI data reliability, the problem is real. Check what you are being sold, though: the underlying job (monitor the data, trace the lineage, page a human) has not changed.

How do you monitor data quality in a RAG pipeline?

Stage by stage, with a check at each handoff rather than one check at the end: source, ingestion, chunk and parse, embed, index, retrieve. Every stage can succeed technically while destroying the data, so a green run proves nothing.

One aside. If part of your corpus comes from the public web (competitor docs, regulatory sites, vendor changelogs), the crawl that turns those pages into clean, structured text is itself a data source that can break silently: a site redesign changes the DOM, the extractor starts returning nav menus instead of body copy, and the corpus fills with cookie banners. Monitor its output volume and text quality, not just HTTP status codes.

Why evals are not enough on their own

An eval suite runs a fixed set of questions on a schedule or in CI, and tells you quality regressed. It does not tell you why, it does not run continuously against production data, and its curated set may never touch the corner of the corpus that broke.

An eval score dropping four points on Tuesday is a symptom. A freshness alert saying the Confluence sync last landed rows 62 hours ago, with lineage showing which index and which agent depend on that table, is a diagnosis. Run both. Evals catch quality regressions at test time. Pipeline monitoring catches the job that broke at 3am, the one that reaches a customer before anyone opens a laptop.

Upstream, where the tables feeding your features and documents are governed by data contracts, a schema change becomes a blocked pull request instead of a page. Contracts prevent what you can specify. Monitoring catches the rest.

What should I do this quarter?

Pick the one AI feature customers see. Write down its dependency chain, source to index, and put a check on every arrow. Start with three: freshness on the source tables, re-embedding lag with a stated SLA, and index-to-source reconciliation. Route the alerts to a channel the on-call person reads, with an owner attached.

None of this needs a new category of tool. An AI pipeline is a data pipeline with two extra stages, and the five pillars still hold. Our platform connects read-only to Snowflake, BigQuery, Databricks, or Redshift, is dbt-native, does column-level lineage, and alerts to Slack and PagerDuty, with setup taking about 15 minutes. To see what breaks in your own pipeline first, pricing starts at $99 a month with a 14-day free trial. If you are still comparing, our rundown of data quality tools and our data quality monitoring guide say where we are the wrong fit.

The unglamorous version of AI trust is a freshness check on a table nobody thinks about. Start there.