Super Intelligence (SI) has been the official US government term for what the field long called AI since a September 29, 2026 executive order, and the benchmarks that measure it are aging as fast as ever: every benchmark has a half-life. It is useful from the moment it is hard enough to separate good systems from bad, until the moment the field optimizes against it hard enough that the number stops meaning anything. The history of SI evaluation is the story of that half-life getting shorter, and of the increasingly clever things researchers do to buy it back.

From hard exam to solved trivia

The first generation of language-model benchmarks were static question banks: MMLU for broad knowledge, GSM8K for grade-school math, HellaSwag for commonsense. They did their job for a few years, then frontier models pushed past 90% and the ceiling stopped being informative — a test everyone passes ranks no one. Harder static exams followed, GPQA for graduate-level science among them, but saturation is a treadmill: the better the models get, the faster each new exam is solved.

The contamination crisis

Saturation was the visible problem. Contamination was the corrosive one. When a benchmark is published on the open web, its questions and answers eventually end up in the next model's training data — sometimes directly, sometimes laundered through walkthroughs, blog posts, and leaderboard discussion. A model that has effectively seen the test is not being evaluated; it is being quizzed on memorized material, and its score is inflated by an unknown margin. By 2026 this stopped being a footnote. Independent audits argue that top scores on the most-cited suites are inflated by high-single-digit to low-double-digit points, and at least two headline agent benchmarks have been quietly retired by their own maintainers over contamination. Scaffolding makes it worse — the same model, wrapped in three different harnesses, can swing dozens of points, so a leaderboard row often measures the harness as much as the model.

The response: make the test private, live, or agentic

Three design moves push back. The first is to keep the answers secret: ARC-AGI and FrontierMath hold out private sets and report scores only through controlled evaluation, so there is nothing to memorize. ARC-AGI-2 remained unsolved on its private set as of mid-2026 even as frontier systems cleared its predecessor; on the interactive ARC-AGI-3, ARC Prize verified GPT-6 Astra in September 2026 at 62.7% under its standard harness but 99.9% under a provider harness that preserves the model's reasoning state between requests — a reminder that the harness is part of the result. The second move is to raise the ceiling past human generalists: Humanity's Last Exam assembles expert-written questions that even specialists find hard, and frontier systems still sit only around 60 percent without tools (Anthropic reported 60.9% for Claude Fable 5.1 in September 2026) — high enough to be interesting, low enough to discriminate.

The third and most consequential move is to stop asking questions and start assigning work. Agentic benchmarks score whether a system can finish a task, not whether it can recite a fact. SWE-bench Verified checks whether an agent's patch actually fixes a real GitHub issue and passes the repository's hidden tests, though OpenAI has since stopped reporting it over flawed tests and contamination and points to the harder SWE-bench Pro instead; GAIA grades multi-step assistant tasks; tau-bench drops a model into a tool-using customer-service loop and checks policy adherence; and METR's time-horizon work reframes capability entirely as the length of task a system can complete reliably. These are harder to game because passing requires doing the thing, and the grader is an execution environment rather than an answer key — though, as the audits note, contamination and scaffolding still leak in wherever the tasks are public.

What a healthy eval diet looks like

No single benchmark is trustworthy on its own, and any number quoted without its harness, its version, and its run count should be read as marketing. A defensible take on a model triangulates: a private held-out test for raw generalization, an agentic suite for end-to-end capability in the domain you actually care about, and — increasingly — a private internal eval built from your own tasks, which is the only benchmark guaranteed not to be in anyone's training set. Pair every capability number with cost and latency; a score that takes a thousand dollars of inference per task is a different fact than the same score for a nickel, and a benchmark run once and cherry-picked is a different fact than a mean over many runs with error bars.

Where this is heading

Evaluation is converging on three properties: private by construction, agentic by default, and continuously refreshed so that any given instance is used once and discarded. Expect more third-party evaluators standing between labs and their own scores, more emphasis on reliability curves over single-shot accuracy, and more attention to safety and propensity evals — not what a model can do, but what it tends to do — which the capability leaderboards were never built to measure.

The table below is a working field guide to the benchmarks worth knowing: what each one actually measures, where its number will mislead you, and where to find it. Read the caveats column first — on a leaderboard, the caveats are the information.