REDSHIFT DATA QUALITY
Redshift Data Quality: AWS Glue Data Quality, Data Quality Monitoring, and Redshift Observability Compared
Data quality monitoring on every Redshift table, watching freshness, volume, schema, and distribution, with no ruleset to author per table and no Glue crawler to keep in sync. Read only connection, alerts in Slack and PagerDuty, from 99 dollars a month.
14-day trial, no credit card, read-only connection
Alerted #data-eng 0.8s ago.
Downstream impact · consumers at risk
How do you monitor data quality in Amazon Redshift?
Redshift has no built in data quality engine of its own. The native route is AWS Glue Data Quality, which evaluates DQDL rules against a Redshift table that has been cataloged in the AWS Glue Data Catalog, recommends rules for you, and since July 2026 can run ML anomaly detection on cataloged tables when you start an evaluation run with ObservationScope set to ALL. Underneath that, Redshift system views carry the raw signal: SVV_TABLE_INFO gives row estimates without scanning, and SYS_QUERY_HISTORY and SYS_QUERY_DETAIL show what actually loaded. CloudWatch watches the cluster, not the data in it. The gaps are consistent: rulesets are authored per table, Data Catalog evaluation runs cannot identify which rows failed, statistics are capped at 100,000 per account, and alerting stops at CloudWatch and EventBridge. Dataobservability connects to Redshift read only, learns each table baseline from its own history, and alerts on all four pillars across every schema, from 99 dollars a month.
Last updated August 2026
Side by side
Redshift data quality compared
Swipe to see all columns →
| Route to Redshift data quality | What it catches | What you build and operate | Documented limits (AWS docs, August 2026) |
|---|---|---|---|
| AWS Glue Data Quality via the Data Catalog | DQDL rules you authored or accepted from recommendations, plus a data quality score per run | A cataloged table per Redshift table, a ruleset, an IAM role, an EventBridge schedule, and somewhere to send results | Identifying the records that failed a check is not supported for Data Catalog runs, only for Glue ETL jobs. Rule recommendation runs are deleted after 90 days. Rules cannot evaluate nested or list type data. Athena views cataloged in Glue are not supported |
| AWS Glue Data Quality anomaly detection | Statistical outliers in gathered statistics such as Completeness, Uniqueness, Mean, Sum, StandardDeviation, Entropy, DistinctValuesCount, and UniqueValueRatio | Analyzers on the columns you nominated, evaluation runs on a schedule, and feedback on every observation to keep the model honest | Needs a minimum of three data points before it can detect anything. A detected anomaly is treated as a normal value in later runs unless you explicitly exclude it. Anomalies do not affect the data quality score, so a passing score can sit on top of an open observation |
| DQDL rulesets | Row and column level assertions such as IsComplete, IsUnique, ColumnValues, ColumnLength, and ReferentialIntegrity across datasets | Every rule, per table, kept in step with every schema change your upstream teams make | A ruleset is capped at 2,000 rules and 65KB, and must be associated with a table in the Glue Data Catalog. Preprocessing queries are API only, are not available in the console, and cap at 51,200 characters |
| Redshift system views | Raw evidence: SVV_TABLE_INFO for row estimates without a scan, SYS_QUERY_HISTORY and SYS_QUERY_DETAIL for what actually ran | The queries, the schedule, a results table, the baselines, the thresholds, the deduplication, and the alerting | These are views, not a monitoring product. Redshift has no native anomaly detection and no native column level lineage graph, so downstream impact has to be reconstructed from query text |
| Amazon CloudWatch | Cluster and workload health: CPU, storage, connections, query duration, WLM queue depth | Alarms and thresholds per metric | Watches the cluster, not the correctness of the data inside it. A cluster can be perfectly healthy while a nightly load silently delivers half the rows |
| Dataobservability | Freshness, volume, schema, and distribution breaks across every Redshift table, with lineage and grouped incidents | A read only connection, no agent | Covers the connected warehouses (Snowflake, BigQuery, Databricks, Redshift) and dbt. Systems outside them are not monitored |
Positioning and pricing models are summarized in good faith from each vendor's public pages, August 2026. Verify current terms with the vendor.
What you get
What a monitoring layer adds on top of Glue Data Quality
Coverage that does not depend on the Glue Data Catalog being current
Every native Redshift quality check runs through a table that has been cataloged in Glue first. That is a second inventory to keep in step with the first: a crawler or a registration step per table, and a ruleset attached to it. The failure mode is quiet, because a table that never made it into the catalog is not reported as unmonitored, it simply does not appear. Metadata driven monitoring reads the Redshift catalog itself, so the table list is the input rather than something a crawler is supposed to have noticed last night.
Row level answers on the tables you actually query
AWS documents the split clearly: identifying which records failed a data quality check is supported in Glue ETL jobs and not supported for Data Catalog evaluation runs. If your Redshift tables are monitored the Data Catalog way, which is the way most teams start because it needs no code, a failed rule tells you the table is wrong and not which rows made it wrong. Getting to the offending records means rebuilding the check as an ETL job, or writing the query by hand at the moment you least want to.
Schema drift on every column, not the ones a rule already named
DQDL is a good assertion language and it only ever knows about columns somebody wrote a rule for. A vendor adding a column, an upstream job retyping an amount from numeric to varchar, or a dropped field that a downstream view quietly resolves to null all pass a ruleset that never mentioned them. Watching the catalog for adds, drops, and type changes is a different mechanism from asserting values, and it is the one that catches the breakages nobody predicted.
Statistics that are not capped or aged out from under you
Glue Data Quality stores the statistics it gathers for free, with a documented limit of 100,000 statistics per account and a maximum retention of two years. On a wide Redshift estate with per column analyzers, that account level ceiling is closer than it sounds, and it is shared with every other Glue workload you run. History is what makes a baseline trustworthy, so a monitoring layer that keeps its own metric history per table does not compete with your ETL jobs for the same allowance.
Alerts routed to an owner instead of into CloudWatch
Native results publish metrics to CloudWatch and key events to EventBridge, which is the correct place for them and not an on call path. Getting from an EventBridge event to the engineer who owns the dataset means a rule, a target, a lookup of who owns what, and state so a table that has been stale for three days does not fire on every scheduled evaluation. Routing per dataset owner into Slack and PagerDuty with deduplication already handled is the difference between a signal and a filter rule.
Baselines that know a table is quiet at the weekend
Most false alarms are calendar failures rather than statistical ones. A table loaded hourly on weekdays and never on Sunday is not broken on Sunday, and a fixed DQDL threshold cannot tell the difference. Glue anomaly detection does model this, and its Linear mode is explicitly built for trends and seasonality, but it needs at least three data points per statistic and analyzers on the columns you chose in advance. Baselines derived automatically from each table's own load history cover the tables nobody nominated too.
How it works
From connected to caught
Connect Redshift with a read only user
Create a user with USAGE on the schemas you want watched and SELECT on the tables, plus access to the system views. Dataobservability reads the Redshift catalog, SVV_TABLE_INFO statistics, and query history to build the inventory and the freshness and volume picture, so table contents are not copied out of your cluster and the scans that cost you money do not run. There is no agent, nothing to install in the VPC, and no change to existing jobs.
Let baselines build before you touch a threshold
Give it a few days of load history per table so weekly seasonality becomes visible. Tuning thresholds before a baseline exists is how teams manufacture alert fatigue in week one and stop trusting the monitoring by week two. If you already run Glue Data Quality, leave it running during this period and compare what each one flags. They fail differently, which is the useful part.
Keep DQDL for the invariants, not the general shape
Rules are good at what a learned baseline cannot know: that a discount percentage never exceeds 40, that every order carries a valid region code, that a foreign key resolves in another dataset. DQDL covers that last one well with ReferentialIntegrity and DatasetMatch. Use rules there and leave outages, drift, and distribution shifts to baselines. That split is what keeps a ruleset finite instead of growing until nobody dares delete a line, which matters when the cap is 2,000 rules and 65KB.
Route by schema owner, then prune what never fires
Send alerts to the team that owns each schema rather than into one shared channel. After two weeks, read the log back: a monitor that has never fired on a real problem should be relaxed, and any incident that reached you through a complaining stakeholder should become a monitor. Monitoring that is never pruned turns into noise, and noise gets muted, usually by the person you most needed to reach.
AWS Glue Data Quality is the native answer, and it is a rule engine
Redshift itself ships no data quality feature. There is no equivalent of Snowflake data metric functions or Databricks Unity Catalog anomaly detection sitting inside the warehouse. What AWS provides instead is AWS Glue Data Quality, a managed serverless service built on the open source DeeQu framework, which evaluates rules written in DQDL, the Data Quality Definition Language. Redshift support arrived with general availability, and the mechanism is worth being precise about: Glue Data Quality evaluates objects stored in the AWS Glue Data Catalog, so a Redshift table is covered once it is cataloged, alongside Amazon S3, other JDBC sources, and the transactional lake formats. There are two entry points into the service and they are not interchangeable. Data quality for the Data Catalog is the no code path, aimed at data stewards and analysts, and it supports rule recommendations: point it at a table, choose Recommend rules, and it analyzes each column and proposes DQDL such as IsComplete on a column that is never null, Uniqueness above a threshold on something that looks like a key, ColumnValues restricted to the set of values it observed, and ColumnLength bounded by the lengths it saw. Data quality for Glue ETL jobs is the code path, evaluated inside a job as data moves, which is where you filter bad records out before they land. The feature matrix between the two has real consequences for a Redshift team. Rule recommendations are supported for the Data Catalog and not supported for ETL jobs. Identifying the records that failed a data quality check is supported for ETL jobs and not supported for the Data Catalog. Auto scaling and Glue Flex are ETL only. So the path that is easiest to adopt on a Redshift table is also the one that will not tell you which rows broke, and the path that tells you which rows broke requires you to move the data through a Glue job you write and operate.
Anomaly detection, what it genuinely does, and the two footnotes that matter
Anomaly detection is the part of Glue Data Quality that behaves least like a rule engine, and AWS has been steadily widening it. It went generally available for Glue ETL in August 2024, and on July 27, 2026 AWS extended it to the Data Catalog: you enable it for cataloged tables by starting an evaluation run with ObservationScope set to ALL. That matters for Redshift specifically, because the Data Catalog is how Redshift tables get covered in the first place. The mechanism is straightforward. Alongside rules you configure analyzers, which gather statistics without asserting anything, using expressions like AllStatistics on a column, DistinctValuesCount on another, or a bare RowCount. Glue stores those statistics over time, and a machine learning model learns the trend and predicts a range for the next value. When an actual value lands outside the predicted range, Glue raises an Anomaly Observation, showing the actual trend, a derived trend, upper and lower bounds, and recommended DQDL rules that would catch the same issue in future. It captures seasonality without configuration, so weekday and weekend patterns are learned rather than declared, and since July 2026 there is a Distribution analyzer that computes binned histograms for numeric columns and frequency sorted value distributions for categorical ones. Two modes control the forecast: Linear, the default, models trends and seasonality and suits regular schedules, while Fixed treats all data points as equally spaced and suits flat or irregularly evaluated data. Now the footnotes. First, it needs a minimum of three data points before it can detect an anomaly, so a new table is unmonitored for its first three evaluation runs, and on a daily schedule that is three days. Second, and this is the one that surprises people in production: when an anomaly is detected it is considered normal for subsequent runs, and the algorithm will treat that anomalous value as input unless it is explicitly excluded. In other words the model quietly learns your outage. AWS is direct that retraining through acknowledging or rejecting observations is critical, which means anomaly detection here is not a set and forget feature, it is a feedback loop somebody has to work. Third, anomalies do not affect the data quality score, so a table can report a healthy score while carrying an open observation. That is defensible design, and it does mean the score alone is not a health indicator.
The limits you will actually hit on a Redshift estate
The documented service limits are the best guide to how far the native route scales, and there are four worth writing down before you plan around it. A ruleset can hold 2,000 rules, and the ruleset itself is capped at 65KB, with AWS recommending you split larger ones into multiple rulesets. That is generous per table and becomes an inventory problem across hundreds of tables, since each ruleset is an object attached to one cataloged table that someone maintains. Glue Data Quality collects statistics whenever you create a rule or an analyzer, storage of those statistics is free, and there is a limit of 100,000 statistics per account with a maximum retention of two years. Both halves of that matter: the cap is per account rather than per table, so a wide estate with per column analyzers shares it with every other Glue workload, and two years is the ceiling on how much history a baseline can ever draw on. Rule recommendation runs are automatically deleted after 90 days, so the recommendation you meant to review last quarter is gone. Rules cannot evaluate nested or list type data sources, and AWS points you at flattening the structs first, which is less painful in Redshift than in a lake but still real if you land SUPER columns. Athena views cataloged in the Glue Data Catalog are not supported, and Delta Lake symlink tables are excluded from both recommendation runs and Data Catalog evaluation runs. Preprocessing queries, which let you derive columns or filter before checks run, are supported only through the APIs and not the console, must be valid SELECT statements referencing the table as database.table in backticks, cap at 51,200 characters, and must return at least one row. None of these is a flaw. They are the shape of a rule evaluation service rather than a monitoring system, and they tell you exactly where the build ends.
What the Redshift system views give you, and what they cannot
Underneath Glue, Redshift exposes enough metadata to build a real freshness and volume picture yourself, and teams that go this route are not wrong to. SVV_TABLE_INFO is the important one, because it reports table statistics including row estimates without scanning the table, which means a volume check that costs almost nothing. SYS_QUERY_HISTORY and SYS_QUERY_DETAIL, the monitoring views introduced in January 2025, record what actually ran: the statement, the user, the timing, and the tables touched. From those you can derive when a table was last written, whether the nightly load ran at all, and how long it took, which covers freshness and gives you a foothold on volume. What you cannot get from them is a monitoring system. There is no baseline, so every threshold is a number a person chose and now owns. There is no state, so a table stale since Friday produces an identical alert every run until somebody fixes it. There is no distribution tracking unless you write and store the profiling queries yourself, and that is where warehouse cost quietly appears, since profiling means scanning. Redshift also has no native column level lineage graph. SYS_QUERY_HISTORY and SYS_QUERY_DETAIL give you SQL text, so reconstructing which dashboard depends on which column means parsing query text, and that is a project rather than a query. This is the honest difference against Snowflake, where ACCESS_HISTORY exposes objects_modified with directSources and baseSources on Enterprise Edition, and against Databricks, where Unity Catalog builds column lineage automatically. On Redshift, downstream impact is something you assemble.
Build on the AWS primitives or buy the layer above them
The straight answer is that Glue Data Quality is good enough for a real set of Redshift teams, and if you are one of them you should not pay anyone. If you have a few dozen tables that matter, they are already cataloged in Glue, and an engineer is happy owning rulesets and reviewing observations, run rule recommendations, keep the rules that survive review, enable anomaly detection with ObservationScope ALL, publish to CloudWatch, and stop reading. The line moves when coverage becomes the goal rather than a curated list. At that point you are not writing checks any more, you are operating a system: a catalog registration step for every new table, a ruleset per table maintained against upstream schema changes, EventBridge schedules, a state store so repeat problems do not re alert, an ownership map, routing into Slack or PagerDuty, acknowledgement and resolution, and a feedback loop on every anomaly observation so the model does not learn your outages as normal. Add the cost shape: Glue charges for the time it takes to detect anomalies, at one DPU per statistic, so the bill grows with how many columns you profile and how often. That work is not difficult, which is exactly why it gets underestimated. Teams ship version one in an afternoon and spend two quarters on the parts nobody scoped, funded by the most senior data engineer available and taken straight out of the pipeline work they were hired for. It is also why published pricing matters more in this category than in most. Of the thirteen observability tools we track, three publish a price: Soda at 750 dollars a month for its Team plan, Metaplane, and us. Monte Carlo publishes no list price on its own site, though its AWS Marketplace listing shows a 12 month contract at 50,000 dollars for its credit dimension, with overage at one cent per unit. Bigeye pricing page returns a 404, while its AWS Marketplace listing shows 45,000 dollars a year for 100 actively monitored tables and 75,000 dollars for 300. Datafold redirects its pricing page to a contact form. All three figures were checked directly on the listings in August 2026. Dataobservability is 99, 299, and 799 dollars a month, with a 14 day trial that needs no credit card, so you can hold a real number against the engineering estimate before anyone books a call.
Questions buyers ask
Redshift data quality FAQ
Does Amazon Redshift have built in data quality checks?
No. Redshift ships no data quality engine of its own, unlike Snowflake data metric functions or Databricks Unity Catalog anomaly detection. The native AWS answer is AWS Glue Data Quality, which evaluates DQDL rules against Redshift tables that have been cataloged in the AWS Glue Data Catalog. Redshift itself contributes the raw signal through system views such as SVV_TABLE_INFO, SYS_QUERY_HISTORY, and SYS_QUERY_DETAIL.
What is AWS Glue Data Quality?
It is a managed, serverless data quality service built on the open source DeeQu framework. You express expectations in DQDL, the Data Quality Definition Language, with more than 25 out of the box rule types, and evaluate them either against tables in the AWS Glue Data Catalog or inside Glue ETL jobs. Each evaluation returns a data quality score, which is the percentage of rules in the ruleset that passed.
What is DQDL?
DQDL is the Data Quality Definition Language, the domain specific language AWS Glue Data Quality uses to express rules. A rule is an expression that checks one characteristic and returns a Boolean, for example IsComplete on a column, Uniqueness above a threshold, ColumnValues restricted to a set, or ColumnLength within a range. It also supports cross dataset rules such as ReferentialIntegrity, RowCountMatch, SchemaMatch, and DatasetMatch.
Does AWS Glue Data Quality support anomaly detection?
Yes, in both Glue ETL and the Data Catalog. It went generally available for ETL in August 2024, and on July 27, 2026 AWS extended it to cataloged tables: you start an evaluation run with ObservationScope set to ALL. It needs a minimum of three data points before it can detect anything, and it supports two forecast modes, Linear by default for trends and seasonality, and Fixed for flat or irregularly evaluated data.
Does Glue Data Quality tell you which rows failed a check?
Only in Glue ETL jobs. AWS documents identifying the records that failed data quality checks as supported for ETL jobs and not supported for the Data Catalog. Since cataloging is how Redshift tables are covered, the easiest path to adopt is also the one that reports a failed rule without the offending rows. Getting row level answers means rebuilding the check as a Glue ETL job you write and operate.
How many rules can an AWS Glue Data Quality ruleset have?
A ruleset can hold 2,000 rules and is capped at 65KB in size, and AWS recommends splitting anything larger into multiple rulesets. A ruleset must also be associated with a table in the Glue Data Catalog, so the practical ceiling is rarely the rule count. It is the number of rulesets a team can keep in step with upstream schema changes across hundreds of tables.
How long does AWS Glue Data Quality keep statistics?
There is no charge to store the statistics Glue gathers from rules and analyzers, but there is a limit of 100,000 statistics per account and they are retained for a maximum of two years. The cap is per account rather than per table, so wide estates with per column analyzers share it with every other Glue workload. Rule recommendation runs are separately deleted after 90 days.
Does a detected anomaly affect the AWS Glue data quality score?
No, and this catches people out. AWS documents that when anomalies are generated the data quality score is not impacted, so a table can report a healthy score while carrying an open anomaly observation. A second behavior matters just as much: once an anomaly is detected it is treated as a normal value in later runs unless you explicitly exclude it, so the model learns your outage unless somebody rejects the observation.
Can I detect schema changes in Redshift automatically?
Not through DQDL alone, because a rule only ever knows about columns somebody named. SchemaMatch compares a dataset against a reference, which helps where you maintain one. Catching an added, dropped, or retyped column across every table is a catalog watching mechanism rather than an assertion, which is what an observability layer runs continuously against the Redshift catalog and then maps to the downstream models and dashboards affected.
Does Redshift have column level lineage?
No. Redshift exposes no native column level lineage graph. SYS_QUERY_HISTORY and SYS_QUERY_DETAIL record the SQL that ran, so lineage can be reconstructed by parsing query text, but that is a project rather than a query. This is a genuine difference from Snowflake, where ACCESS_HISTORY exposes objects_modified with directSources and baseSources on Enterprise Edition, and from Databricks, where Unity Catalog captures column lineage automatically.
More of the platform
Catch broken data before your stakeholders do
Connect your warehouse and get redshift data quality live from one read-only connection. Transparent pricing, no credit card.