BIGQUERY ANOMALY DETECTION
BigQuery Anomaly Detection: BigQuery Data Quality and Data Warehouse Observability Monitoring
Anomaly detection across every BigQuery table, watching freshness, volume, schema, and distribution, with no model to train and no rules to author per table. Read only connection, alerts in Slack and PagerDuty, from 99 dollars a month.
14-day trial, no credit card, read-only connection
Alerted #data-eng 0.8s ago.
Downstream impact · consumers at risk
Does BigQuery have built in anomaly detection?
Yes, but not as a monitoring system. BigQuery gives you detection primitives: AI.DETECT_ANOMALIES runs against the built in TimesFM foundation model, ML.DETECT_ANOMALIES works with ARIMA_PLUS, ARIMA_PLUS_XREG, K-means, autoencoder, and PCA models you create and train yourself, and BigQuery data quality scans in Knowledge Catalog check rules you write per table. All of them return rows. None of them watch your warehouse, decide what is normal for each table, or page the right engineer at 3am, so the scheduling, state, thresholds, routing, and escalation are yours to build and keep running. Dataobservability connects to BigQuery read only, learns each table baseline from history, and alerts on freshness, volume, schema, and distribution breaks across every dataset, from 99 dollars a month.
Last updated August 2026
Side by side
BigQuery anomaly detection compared
Swipe to see all columns →
| How to detect anomalies in BigQuery | What it catches | What you build and operate | Documented limits (Google Cloud docs, August 2026) |
|---|---|---|---|
| AI.DETECT_ANOMALIES with the built in TimesFM model | Anomalies in time series data | The scheduled query, a results table, thresholds, dedupe, and alert routing | Time series data only; a point is flagged when its anomaly probability exceeds ANOMALY_PROB_THRESHOLD, so tuning that threshold per metric is your job |
| ML.DETECT_ANOMALIES with ARIMA_PLUS or ARIMA_PLUS_XREG | Values drifting outside a forecast range | CREATE MODEL and training per time series, retraining as the series changes, plus all of the above | Requires a trained model before you can detect anything; one model per series means model sprawl across a real warehouse |
| ML.DETECT_ANOMALIES with K-means, autoencoder, or PCA | Multivariate outliers in independent, identically distributed data | Model training, feature selection, and a contamination or threshold choice | K-means flags on the shortest of the normalized distances to each cluster centroid; autoencoder and PCA flag on reconstruction loss as mean squared error, neither of which maps cleanly to a business meaning |
| Supervised models scored with ML.PREDICT | Anomaly classes you have already labeled | A labeled training set, plus retraining as new failure modes appear | Needs labeled anomalies. Most teams do not have them, and the failures that hurt most are the ones nobody labeled yet |
| BigQuery data quality scans (Knowledge Catalog, renamed from Dataplex Universal Catalog on April 10, 2026) | Rule breaks: completeness, uniqueness, freshness, validity, custom SQL | One rule set authored and maintained per table | Limited to 1,000 rules per scan; runs on BigQuery and Iceberg REST Catalog tables only; alerting is Cloud Logging plus email to at most five addresses; column level quality scores are available through the API only; rule recommendations are not supported in the gcloud CLI |
| Dataobservability | Freshness, volume, schema, and distribution breaks across every table | A read only connection, no agent | Covers the connected warehouses (BigQuery, Snowflake, Databricks, Redshift) and dbt; systems outside them are not monitored |
Positioning and pricing models are summarized in good faith from each vendor's public pages, August 2026. Verify current terms with the vendor.
What you get
What a monitoring layer adds on top of the BigQuery primitives
Baselines learned per table, not thresholds you guess
The hard part of anomaly detection is never the statistics, it is deciding what normal means for eleven hundred tables that each behave differently. A table loaded hourly on weekdays and never on Sunday is not broken on Sunday. Baselines derived from each table's own history encode that automatically, where a hand written rule has to be told, and then told again every time the schedule changes.
Every table covered on day one, including the ones nobody rules
Rule based checks only cover tables somebody remembered to write rules for, and the table that breaks is reliably the one nobody thought about. Metadata driven monitoring starts from the full table list, so a dataset added last week is watched from the moment it appears rather than from the moment an engineer gets around to it.
Alerts that reach the people who carry the pager
Native BigQuery quality scans notify through Cloud Logging and email to at most five addresses. That is a reasonable audit trail and a poor incident channel. Routing anomalies into Slack and PagerDuty, with ownership per dataset, is the difference between a signal that gets acted on and one that fills an inbox nobody reads on a Saturday.
No model to train, retrain, or explain
BQML anomaly detection means CREATE MODEL, a training run, and a retraining story for every series you care about. That is tractable for the five metrics on the executive dashboard and unmanageable across a warehouse. It also leaves you explaining reconstruction loss to a stakeholder who asked why the revenue number looked wrong.
Query cost kept low by reading metadata first
The naive way to monitor a warehouse is to scan tables on a schedule, which turns monitoring into one of your larger BigQuery line items. Freshness and volume checks answer from table metadata and statistics rather than full scans, so coverage grows with the number of tables without the bill growing the same way.
Anomalies arrive with downstream impact attached
A detected outlier is a fact. What an on call engineer needs is the blast radius. Because monitoring and column level lineage sit in the same system, a distribution break on a source field arrives with the specific downstream models and dashboards that read it, rather than sending someone to a second tool to work it out by hand.
How it works
From connected to caught
Connect BigQuery with a read only service account
Grant a service account read access to the datasets you want watched. Dataobservability reads INFORMATION_SCHEMA, table metadata, and job history to build the table inventory and the freshness and volume picture, so table contents are not copied out of BigQuery. Setup is a read-only connection with no agent and no pipeline changes.
Let baselines build before you tune anything
Give it a few days of history per table so the seasonality is visible: weekday and weekend load patterns, month end spikes, the batch that lands at 04:15. Tuning thresholds before the baseline exists is how teams end up with alert fatigue on day three and monitoring nobody trusts by week two.
Add rules only where the business logic is not statistical
Baselines catch the drift and the outages. They cannot know that a discount percentage must never exceed 40, or that every order must map to a valid region code. Write custom SQL checks for those specific invariants and leave the general shape of the data to the learned baselines. The mix is what keeps rule maintenance finite.
Route by dataset ownership, then start reviewing what fired
Send alerts to the team that owns each dataset in Slack or PagerDuty rather than to one shared channel. After the first two weeks, read back through what fired: the monitors that never fire on a real problem should be relaxed, and the incidents that arrived by way of a complaining stakeholder should become monitors.
What BigQuery anomaly detection actually gives you, and what it leaves for you to build
Search for BigQuery anomaly detection and you land on BigQuery ML, which is a genuinely capable set of tools that is also frequently mistaken for something it is not. Two functions do the detection. AI.DETECT_ANOMALIES runs against the built in TimesFM foundation model, so you can detect anomalies in a time series without creating and training a model first. ML.DETECT_ANOMALIES covers the models you build yourself: ARIMA_PLUS and ARIMA_PLUS_XREG for time series, and K-means, autoencoder, and PCA for independent, identically distributed data. There is also a supervised route, where you train a classifier such as boosted trees, a random forest, or a deep neural network on labeled anomalies and score new rows with ML.PREDICT. The mechanics differ in ways worth knowing. For time series, a value is flagged as anomalous when its anomaly probability, computed from the actual value against the predicted value and the variance learned during training, exceeds the ANOMALY_PROB_THRESHOLD you set. For K-means, a point is flagged based on the shortest of the normalized distances from it to each cluster centroid. For autoencoder and PCA, the signal is reconstruction loss expressed as mean squared error. Each of those is a defensible statistical definition of unusual, and none of them is a definition of broken. That gap is the whole issue. A SQL function that returns rows where is_anomaly is true is a detector. A monitoring system is a detector plus a schedule, plus somewhere to store what was already seen so the same anomaly does not alert twelve times, plus per table thresholds, plus ownership, plus routing into the channel where your on call engineer actually lives, plus a way to acknowledge and resolve. Everything after the word detector is what you inherit when you decide to build this on BQML, and it is considerably more work than the query that started the project. Teams routinely get the first version running in an afternoon and then spend the next two quarters on the parts nobody scoped.
The other native option: BigQuery data quality scans, and the ceilings you should know before adopting them
The rule based path is BigQuery data quality scans, powered by Knowledge Catalog, which was renamed from Dataplex Universal Catalog on April 10, 2026. This is closer to a monitoring product than BQML is. You define rules against a table and the scan runs on a schedule, hourly, daily, weekly, monthly, or on a cron expression, with on demand and one time runs also available. Rule types cover the dimensions you would expect: completeness for nulls, uniqueness, validity through range checks, set checks, and regular expressions, freshness against a timestamp column, and statistic checks on aggregates such as mean, min, and max. Custom SQL rules handle row conditions, table conditions, and assertions when the built in types are not enough. Scans can sample rather than read everything, at any percentage between 0.0 and 100.0 with up to three decimal places. It is a real tool and for a bounded set of critical tables it is a sensible choice. Four documented ceilings tend to decide whether it scales to your situation. First, a data quality scan is limited to 1,000 rules, which sounds generous until you multiply a handful of rules by every table in a mature warehouse. Second, rules run on BigQuery and Iceberg REST Catalog tables only. Third, alerting is Cloud Logging plus email notifications to at most five addresses, which is an audit trail rather than an incident channel, and the auto data quality documentation does not offer Pub/Sub for these alerts, so getting to Slack or PagerDuty means building the bridge yourself through logs. Fourth, column level data quality scores are exposed through the API only, so the granular view you want during triage is not in the console. There is also the authoring cost, which is the one that quietly dominates. Every rule is a statement about one column on one table, written by a person who understood that column on the day they wrote it. Warehouses change. Rule sets rot, and the rot is invisible until the day a rule that has been silently passing against a renamed column fails to catch a real incident.
Cost anomalies and data anomalies are different problems, and the search results mix them together
A large share of what gets written about BigQuery anomaly detection is really about billing. Detecting that a runaway query burned through slots overnight, or that on demand spend tripled after somebody removed a partition filter, is a legitimate and painful problem, and the standard recipe is to export billing data to BigQuery, fit an ARIMA_PLUS model on daily spend, and wire alerting policies to the result. If your spend has been surprising you, that pipeline is worth building or buying, and the broader discipline of watching cloud and SaaS spend for unexpected movement is its own category with its own tooling, kept separate from data monitoring. It is worth being clear that this is a different job from the one this page is about. Cost anomaly detection watches one time series, your spend, and the consequence of missing it is a bill. Data anomaly detection watches thousands of tables and the consequence of missing it is that a customer facing number was wrong for a week and nobody knew. The signals are different, the cardinality is different by three orders of magnitude, and the remediation is different. Teams that build the cost pipeline first sometimes assume they have solved the data problem with the same pattern, and then discover that a technique which works cleanly on one well behaved daily series does not survive contact with eleven hundred tables that each load on their own schedule. Worth noting too: a table can be perfectly fresh and perfectly sized and still be wrong. Volume and freshness catch outages. Distribution monitoring is what catches the silent case, where a currency field starts arriving in cents instead of dollars, or a categorical column picks up a new value that every downstream case statement quietly drops into an else branch.
What BigQuery anomaly detection costs, counting the part that is not on the invoice
The direct costs are reasonably transparent, which is unusual for this category. BQML anomaly detection bills as BigQuery compute: model creation and training consume resources, and so does every scheduled inference query you run, which means the cost scales with how many series you watch and how often you check them. BigQuery data quality scans bill on DCU hours through the Knowledge Catalog premium SKU, metered per second with a one minute minimum, and if you specify a custom execution identity the compute and storage are billed directly to your BigQuery project instead, which lets you apply existing slot commitments and enterprise discounts. Neither is expensive for a small footprint, and you should price your own case rather than take a figure from a blog, because the charge depends on rows, columns, bytes scanned, partitioning and clustering, and scan frequency. The cost that decides most build versus buy arguments is not on any invoice. It is an engineer, indefinitely. Someone writes the detection queries, someone schedules them, someone builds the state table so alerts deduplicate, someone tunes thresholds per table after the third false alarm in a week, someone maintains the rules as columns are renamed, someone builds the Cloud Logging bridge to Slack, and someone is responsible when the monitoring itself silently stops running. In practice this lands on the most senior data engineer available, and it competes directly with the pipeline work they were hired for. Half of that person for a year costs multiples of any tool in this category. That comparison is the honest one, and it is also why published pricing matters so much here. Only three of the thirteen observability tools we track publish a price at all: Soda, Metaplane, and us. Monte Carlo does not, Bigeye pricing page returns a 404, Datafold redirects pricing to a contact form, and the enterprise catalogs quote rather than publish. Dataobservability is 99, 299, and 799 dollars a month, with a 14 day trial that needs no card, so you can put a real number against the engineering estimate before anyone writes a business case.
How to evaluate anomaly detection on BigQuery without getting fooled by the trial
Anomaly detection demos are misleading in a specific and predictable way. On sample data every tool looks precise, because sample data has clean seasonality, no backfills, no late arriving partitions, and no table that legitimately goes quiet for three days at quarter end. Your warehouse has all of those, and they are exactly what generates false positives. So bring your awkward tables to the trial and check them first rather than last. Pick a table that loads on an irregular schedule, one that gets backfilled, one that is partitioned by ingestion time with late data, and one small dimension table that barely changes. If a tool floods you on those four, it will flood you on the other eleven hundred, and a monitoring system people mute is worse than no monitoring system because it produces the feeling of coverage without the fact of it. Then test the two things vendors rarely demo. First, silent failures: deliberately break something subtle rather than obvious. Change a currency scale, introduce a duplicate key, let a categorical column pick up a value nobody planned for. Freshness and volume monitoring will not catch any of those, and the distinction between a tool that catches them and one that does not is the distinction between distribution monitoring that works and a marketing claim. Second, time to first alert that matters, measured from signing up rather than from the end of onboarding. A tool that needs six weeks of configuration before it is useful has a real cost that is invisible in a feature comparison. Finally, ask the questions with numbers in the answers. Which BigQuery metadata does it read, and does it scan table contents. How does it handle a table with no load in three days that is supposed to be that way. What is the price, in writing, for your table count. Anyone who cannot answer the last one in the first call is telling you something about the procurement cycle ahead.
Questions buyers ask
BigQuery anomaly detection FAQ
Does BigQuery have built in anomaly detection?
Yes, as functions rather than as a monitoring product. AI.DETECT_ANOMALIES detects anomalies in time series using the built in TimesFM model, and ML.DETECT_ANOMALIES works with ARIMA_PLUS, ARIMA_PLUS_XREG, K-means, autoencoder, and PCA models you train yourself. Both return rows. Scheduling, per table thresholds, alert deduplication, ownership, and routing to Slack or PagerDuty are not included and are what you build around them.
How do I detect anomalies in BigQuery?
There are three native routes. Run AI.DETECT_ANOMALIES against the built in TimesFM model for time series with no training step. Train a model with CREATE MODEL and score it with ML.DETECT_ANOMALIES, using ARIMA_PLUS for time series or K-means, autoencoder, or PCA for other data. Or define rule based checks as BigQuery data quality scans in Knowledge Catalog. The first two find statistical outliers, the third checks rules you write.
What is the difference between AI.DETECT_ANOMALIES and ML.DETECT_ANOMALIES?
AI.DETECT_ANOMALIES uses the built in TimesFM foundation model, so you can detect anomalies in a time series without creating and training your own model first. ML.DETECT_ANOMALIES scores models you have already trained: ARIMA_PLUS and ARIMA_PLUS_XREG for time series, and K-means, autoencoder, or PCA for independent, identically distributed data. In short, one skips the training step, the other requires it but gives you control over the model.
Can BigQuery detect data quality issues automatically?
Partly. BigQuery data quality scans in Knowledge Catalog run rules on a schedule and cover completeness, uniqueness, freshness, validity, and custom SQL. The automatic part is the execution, not the definition: you author the rules per table. Scans are limited to 1,000 rules, run on BigQuery and Iceberg REST Catalog tables only, and alert through Cloud Logging plus email to a maximum of five addresses.
How much does BigQuery anomaly detection cost?
BQML anomaly detection bills as BigQuery compute for model training and for every scheduled inference query, so cost scales with how many series you watch and how often. Data quality scans bill on DCU hours through the Knowledge Catalog premium SKU, per second with a one minute minimum, or directly to your BigQuery project if you set a custom execution identity. The larger cost is usually engineering time to build and maintain the surrounding system.
Does BigQuery alert you when a table stops updating?
Not on its own. You can approximate it with a freshness rule in a data quality scan against a timestamp column, which sends email to up to five addresses and writes to Cloud Logging, or by querying INFORMATION_SCHEMA for last modified times on a schedule you build. Neither learns each table's normal load pattern, so a table that legitimately pauses at weekends needs an exception written by hand.
What is the best anomaly detection tool for BigQuery?
It depends on scale and on who maintains it. For a handful of critical time series with an engineer who enjoys the work, BQML plus scheduled queries is genuinely fine. For a few dozen tables with clear business rules, BigQuery data quality scans are a reasonable fit within the 1,000 rule ceiling. Past that, a dedicated observability platform that learns baselines across every table and routes to Slack or PagerDuty costs less than the engineering time it replaces.
Can I use BigQuery ML for data quality monitoring?
You can, with a caveat about what BQML is optimized for. It will tell you that a value is statistically unusual against a trained model. It will not tell you that a value is wrong, and the two diverge often: a legitimate marketing campaign looks anomalous, while a currency field silently switching from dollars to cents may not. BQML works best on a small set of well understood numeric series, not as warehouse wide coverage.
How do I monitor BigQuery data freshness?
The cheapest reliable signal is table metadata rather than table contents. Read last modified timestamps from INFORMATION_SCHEMA, or set a freshness rule in a data quality scan against a timestamp column. The part that decides whether it is useful is the expectation: a fixed threshold such as six hours produces weekend false alarms on a weekday only pipeline, so freshness monitoring is only quiet when the expected interval is learned per table.
What is a BigQuery data quality scan?
A scan is a set of rules attached to a BigQuery or Iceberg REST Catalog table, executed on a schedule by Knowledge Catalog, which was renamed from Dataplex Universal Catalog on April 10, 2026. Rules cover completeness, uniqueness, freshness, validity, statistic checks, and custom SQL. Scans can sample between 0.0 and 100.0 percent of rows, are limited to 1,000 rules, and report through quality scores, Cloud Logging, and email.
More of the platform
Catch broken data before your stakeholders do
Connect your warehouse and get bigquery anomaly detection live from one read-only connection. Transparent pricing, no credit card.