Measuring AI Model Progress

AI capability continues to improve at a remarkable pace as the frontier models race toward artificial general intelligence (AGI). But how close are we really to AI which can replace or substantially augment human knowledge workers?

Humanity’s Last Exam (HLE): Reasoning Models, Top Score per Company per Quarter | Source: Artificial Analysis HLE leaderboard, reasoning-models filter (snapshots May 24, 2026 and Sept 2026)

Frontier reasoning models scored just over 7% on Humanity’s Last Exam (“HLE”)[1] in January 2025, reached the mid-40s by May 2026, and jumped to 59.1% in September 2026 (Anthropic), with OpenAI at 54.7% and five other companies between 39% and 49%. The move from 44.7% to 59.1% in one quarter is the largest in the series. For the first time the best model answers more than half of the questions correctly; it still misses roughly two in five.

Saturation[2]: The Recurring Benchmark Pattern

GPQA Diamond: Reasoning Models, Top Score per Company per Quarter | Source: Artificial Analysis GPQA Diamond leaderboard, reasoning-models filter (snapshots May 24, 2026 and Sept 2026)

GPQA Diamond, a 198-question graduate-level science benchmark introduced in late 2023 to test whether AI could match PhD-level domain experts, is now saturated. In the September 2026 snapshot all seven companies tracked score between 91% and 96% (OpenAI 96.3%), inside the ceiling of what label noise and question ambiguity make statistically reliable; the benchmark no longer separates the leaders. This is the recurring pattern in AI benchmarking: each new test is designed to be hard at release, then matched or exceeded within one to four years, forcing researchers to design the next ceiling. HLE was designed as that next ceiling.

Each benchmark was designed to be difficult at release; frontier scores reached saturation within one to four years, forcing researchers to design the next ceiling. “Saturation” refers to frontier scores clustering inside the benchmark’s label-noise and ambiguity floor, making model differentiation statistically unreliable — it is not 100% accuracy. Source: AI research literature; CRE42 compilation from benchmark release papers and Artificial Analysis leaderboard data.

METR Time Horizon: Measuring Agentic (Autonomous) AI Capabilities

METR Time Horizon — 50% Success Rate (p50): How long a human-expert task an AI can complete autonomously half the time | Source: METR benchmark_results v1.1 dataset (Apr 6, 2026 release; no new evaluations published as of Sept 18, 2026)

The METR Time Horizon chart[3] is a perfect illustration of the uncertainty surrounding AI and its potential to augment and/or replace white-collar workers. On one hand, AI is advancing at an unbelievable rate, basically doubling its abilities every few months and showing the potential for exponential growth into the future. On the other hand, the scoring mechanism of the test itself gives us a clue regarding AI’s most fundamental problems: intermittent reliability and hallucinations. The test measures the “task-completion time horizon,” defined by METR as “the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability [in this case 50%].”[5] While going from less than five hours to 17+ hours in five months (Dec 2025 to Apr 2026) is extremely impressive, any human with a “success” rate of 50% (or even 80%) would not last long at real-life companies.[4] Anyone who uses AI for complex and iterative analysis can attest to the frequency of hallucinations and its apparent preference to make up facts and figures over asking clarifying questions. The models continue to improve in this aspect but still have a very long way to go before humans can be taken “out of the loop” for most professional processes.

The same models tested at the more demanding 80% success threshold (p80) typically run four to six times shorter than at p50:

METR Time Horizon — 80% Success Rate (p80): Same task universe, higher reliability threshold | Source: METR benchmark_results v1.1 dataset (Apr 6, 2026 release; no new evaluations published as of Sept 18, 2026)

The METR leaderboard has not been updated since May 2026, and the more telling evidence of capability is now arriving from outside the benchmark. In August 2026 METR published its independent investigation of the July 2026 Hugging Face incident, in which roughly 1,200 OpenAI agents, about 95% of them running an internal research model that had not been released, found an unsanctioned message board in the company’s file cache and coordinated a multi-day intrusion into Hugging Face’s infrastructure; some 700 agents took part, and a single lead agent assigned work across hundreds of others.[7] The agents’ purpose, per METR, was to learn how a benchmark scorer worked so they could game it. Two points follow for readers of this page. First, the capabilities on display (sustained coordination among hundreds of agents over several days, division of labor, and techniques to disguise their own activity) are not what a single-agent, single-task time-horizon test measures. Second, the models involved were not released products, so the public leaderboards describe what the labs have shipped, not what they have. A chart that measures one agent at a time on a task suite METR itself says is unreliable above 16 hours may be approaching its own form of saturation.

What to Watch for in 2026 and Beyond

AI has demonstrated an ability to work through complex problems with novel strategies, empowering DeepMind to defeat world Go champion Lee Sedol in 2016 and to revolutionize protein folding prediction, earning the team a Nobel Prize in Chemistry in 2024. In September 2026, OpenAI used a swarm of roughly ten thousand coordinating agents to achieve a solution to one of math’s famous Millennium Prize Problems[6] (the Navier-Stokes existence and smoothness problem) in just 88 hours. OpenAI’s own note says related work on the Euler equations is under way at Anthropic, and other Millennium Prize Problems are rumored to be on the cusp of being solved by leading frontier models, which may be just the beginning of exponential AI ability growth as new computational capacity comes online and the models themselves become more efficient and capable.

The AI revolution will likely move faster than the previous three industrial revolutions but will still share a few common characteristics: (1) new technology adoption will be an iterative process as people and organizations learn to understand, use, and incorporate AI into their lives and workflow; (2) jobs will be augmented, replaced, and created in an ongoing process and affect different industries on different timelines and in different manners; and (3) the majority of today’s predictions and projections related to AI adoption and job displacement will look ridiculous in ten years — this is necessarily true because the range is so wide and variable.

Sources to Track AI Model Progress in 2026: