AI benchmarks are saturating—and we're running out of ways to measure progress

A systematic arXiv study documents how benchmark saturation is quietly undermining our ability to compare models—and why this matters more than any individual benchmark result.

In plain terms: Models are hitting benchmark ceilings faster than we can introduce new ones. A new academic study systematically maps the saturation problem for the first time—and finds it’s accelerating across most standard evaluations.

The story: Benchmark saturation occurs when top models score so close to the maximum that the benchmark can no longer distinguish between them. The study (arXiv:2602.16763) shows this isn’t limited to a few well-known cases (MMLU, HumanEval)—it’s a structural pattern across the field. New benchmarks typically have a useful lifespan of 12–18 months before leading models cluster at the top.

The implication: announced “improvements” on saturated benchmarks are often meaningless. A model that jumps from 91% to 93% on a benchmark with a 95% ceiling tells you almost nothing about real-world performance differences.

Under the hood: The study introduces a saturation index—the share of top-model scores within one standard deviation of the benchmark ceiling. By this measure, roughly 60% of commonly reported benchmarks are already saturated for frontier models. GPQA and ARC-AGI remain useful. Most MMLU variants do not.

So what: Labs and researchers should publish saturation indices alongside benchmark results. For practitioners evaluating models: saturated benchmarks are marketing, not measurement. Task-specific evaluations on your actual use case remain the only reliable signal.