AWS Glue Data Quality: DQDL Rules and Anomaly Detection

AWS Glue Data Quality is the native way to check data quality on AWS. It is a serverless service built on the open source DeeQu framework, it evaluates rules written in DQDL against tables registered in the AWS Glue Data Catalog or flowing through a Glue ETL job, and it returns a data quality score that is simply the percentage of rules that passed. It also recommends rules for you and, since July 2026, runs machine learning anomaly detection on cataloged tables. This is how each part works, with the documented limits that decide whether it is enough.

If you run Amazon Redshift specifically, the side by side of every native route with its ceiling lives on our Redshift data quality page. This article is the practical walkthrough of Glue itself.

The two entry points are not interchangeable

Everything downstream depends on this choice, so make it deliberately. Glue Data Quality has two entry points, and their feature sets genuinely differ.

Data quality for the Data Catalog evaluates objects already stored in the Glue Data Catalog: Amazon S3, Amazon Redshift, JDBC sources compatible with the catalog, and the transactional lake formats Iceberg, Hudi, and Delta Lake. It needs no code, which is why most teams start here. Data quality for Glue ETL jobs evaluates data as it moves through a job you write, so you can filter bad records out before they land.

Three differences matter in practice:

• Rule recommendations are supported for the Data Catalog and not supported for ETL jobs.
• Identifying the records that failed a check is supported for ETL jobs and not supported for the Data Catalog.
• Auto scaling and Glue Flex are ETL only.

Read the middle one twice. The path that is easiest to adopt is the one that will tell you a rule failed without telling you which rows caused it. That is fine when a check failing means you go re run a load, and painful when it means someone has to reconstruct the query by hand during an incident.

Writing DQDL rules

DQDL, the Data Quality Definition Language, is where you express expectations. A rule is an expression that checks one characteristic of your data and returns a Boolean. There are more than 25 rule types out of the box. A ruleset looks like this:

Rules = [
    IsComplete "order_id",
    IsUnique "order_id",
    ColumnValues "amount" >= 0,
    ColumnValues "region" in ["NA", "EMEA", "APAC", "LATAM"],
    ColumnLength "country_code" = 2,
    Completeness "customer_email" > 0.95,
    RowCount > 1000
]

Dynamic rules are the more interesting form, because they compare against history rather than a number somebody picked in a meeting:

Rules = [
    RowCount > avg(last(10)) * 0.8,
    Sum "amount" > avg(last(5)) * 0.7
]

Cross dataset rules cover the checks a single table cannot answer. ReferentialIntegrity confirms values in the primary dataset exist in a reference dataset, and AggregateMatch, RowCountMatch, SchemaMatch, and DatasetMatch compare two datasets directly. These are genuinely useful and underused, particularly for confirming that a transformed table still reconciles against its source.

Two syntax details save time later. A where clause filters data before a rule applies, so you can check only the current partition. And keywords such as NULL, BLANKS, and WHITESPACES_ONLY let you be explicit about what counts as missing, which matters because ColumnValues will not let NULL values pass during comparisons.

Rule recommendations: a good first draft, not a ruleset

You can generate a starting ruleset without writing anything. Open a table in the Glue console, choose the Data quality tab, then Recommend rules. Glue analyzes each column and proposes DQDL based on what it observes: IsComplete on a column that is never null, Uniqueness above a threshold on something that behaves like a key, ColumnValues restricted to the value set it saw, ColumnLength bounded by observed minimum and maximum lengths.

Treat the output as a draft, because the rules describe the data as it happened to look during that run. A ColumnValues "province" in [...] rule built from a sample will fail the first time a legitimate new value arrives, and a length bound derived from existing rows will reject a longer, perfectly valid one. AWS makes the same point in its own tutorial by walking through which generated rules to relax and which to tighten. One scheduling detail: recommendation runs are automatically deleted after 90 days, so review them while they exist.

Analyzers and anomaly detection: the checks you did not write

Every rule above exists because a person thought of it. Analyzers invert that. An analyzer gathers statistics without asserting anything, which is what you want on columns you know are important but do not yet understand well enough to bound.

Analyzers = [
    RowCount,
    AllStatistics "amount",
    DistinctValuesCount "customer_id"
]

Glue stores those statistics over time and a machine learning model learns the trend, predicting a range for the next value. When an actual value falls outside the predicted range, Glue raises an Anomaly Observation showing the actual trend, a derived trend, upper and lower bounds, and recommended DQDL that would catch the same problem in future. It captures seasonality without configuration, so weekday and weekend patterns are learned rather than declared. Supported statistics include Completeness, Uniqueness, Mean, Sum, StandardDeviation, Entropy, DistinctValuesCount, and UniqueValueRatio. A Distribution analyzer added in July 2026 computes binned histograms for numeric columns and frequency sorted value distributions for categorical ones.

Anomaly detection went generally available for Glue ETL in August 2024, and on July 27, 2026 AWS extended it to the Data Catalog. Enable it by starting an evaluation run with ObservationScope set to ALL:

aws glue start-data-quality-ruleset-evaluation-run \
  --data-source '{"GlueTable": {"DatabaseName": "sales", "TableName": "orders"}}' \
  --role "arn:aws:iam::123456789012:role/GlueDataQualityRole" \
  --ruleset-names '["orders_ruleset"]' \
  --additional-run-options '{"ObservationScope": "ALL", "ObservationMode": "LINEAR"}'

Two modes control the forecast. LINEAR, the default, models trends and seasonality and suits evaluations on a regular schedule. FIXED treats all data points as equally spaced regardless of the real interval between runs, which suits flat data or irregular evaluation.

Now the three behaviors that catch people in production. Anomaly detection requires a minimum of three data points, so a new table is effectively unmonitored for its first three runs. Anomalies do not affect the data quality score, so a table can report a healthy score while carrying an open observation. And most importantly: once an anomaly is detected, it is treated as a normal value in subsequent runs unless you explicitly exclude it. The model learns your outage. AWS is direct that acknowledging or rejecting observations is critical to keeping it accurate, which means this is a feedback loop somebody owns, not a feature you switch on and forget.

The limits worth knowing before you plan around it

The documented service limits are the honest guide to how far this scales.

Two exclusions matter as well. Rules cannot evaluate nested or list type data sources, so SUPER columns and nested structs need flattening first. And Amazon Athena views cataloged in the Glue Data Catalog are not supported, which is awkward given how often the objects analysts query are views.

Cost follows the same shape. You are charged for the time it takes to detect anomalies, at one DPU per statistic, so the bill scales with how many columns you profile and how often you run. That is a line worth watching alongside the rest of your cloud and SaaS spend, because profiling every column on every run is the easy way to turn a cheap feature into an expensive one.

Where Glue Data Quality stops and monitoring begins

Glue Data Quality is a good rule evaluation service, and for a curated set of tables owned by an engineer who reviews observations, it is genuinely sufficient. What it is not is a monitoring system, and the difference is operational rather than technical.

Results publish metrics to Amazon CloudWatch and key events to Amazon EventBridge. That is the correct destination and it is not an on call path. Getting from an EventBridge event to the engineer who owns the dataset needs a rule, a target, a map of who owns what, and state, so a table stale since Friday does not fire an identical alert on every scheduled evaluation. There is no ownership model, no acknowledgement or resolution status, and no deduplication. Coverage is also opt in twice over: a table is only checked if it was cataloged, and only checked properly if someone wrote its ruleset and kept it current.

That is the point where teams start comparing the build against a subscription, and the comparison is usually made against the wrong number. It is not the price of a rule engine, it is the cost of the ownership map, the state store, the routing, the schedules, and the person who reviews every anomaly observation so the model does not learn outages as normal. Our breakdown of data quality management tools covers how the categories differ, and the anomaly detection page covers what learned baselines catch that rules structurally cannot.

Frequently asked questions

Does AWS Glue Data Quality work with Amazon Redshift?

Yes. Glue Data Quality evaluates objects stored in the AWS Glue Data Catalog, and Amazon Redshift is a supported data source there alongside Amazon S3, other JDBC sources, and the transactional lake formats. Redshift support arrived with general availability. The practical requirement is that the Redshift table must be cataloged in Glue before any ruleset can be attached to it.

How is the AWS Glue data quality score calculated?

The data quality score is the percentage of data quality rules in a ruleset that pass, meaning they evaluate to true, when you run an evaluation. It is a straight ratio of passing rules to total rules, so it reflects how many of your assertions held rather than how much of your data was correct. Anomaly observations are excluded from the score entirely.

What is the difference between a rule and an analyzer in AWS Glue?

A rule is an assertion: it checks a characteristic and returns true or false, and it counts toward the data quality score. An analyzer gathers statistics without asserting anything, feeding the anomaly detection model instead. Use analyzers on columns you know matter but cannot yet bound. Glue gathers each statistic only once even when a rule and an analyzer cover the same column.

Can AWS Glue Data Quality replace a data observability platform?

For a curated list of cataloged tables with an owner who reviews observations, yes. For catalog wide coverage it does not, because it has no ownership model, no alert deduplication, no incident state, and no automatic coverage of tables nobody registered. It is the rule and statistics engine. The monitoring layer around it is what you either build or buy.