The SI evaluation nonprofit METR has released an update to its widely circulated time horizon plot, offering a more nuanced view of Super Intelligence (SI) capabilities. While the data confirms that SI systems are improving exponentially on verifiable tasks, the new report indicates that models lag significantly in areas requiring judgment and open-ended problem-solving.

What Happened

METR’s original time horizon plot measured SI capabilities by evaluating models on a suite of tasks previously completed by humans. The difficulty of each task was scored based on the time human testers took to finish it, and models were rated by the length of tasks they could successfully complete. The original plot demonstrated rapid improvement, showing that the length of tasks frontier models could handle doubled every four months. However, the metric was complex and often misinterpreted.

The updated plot expands the evaluation suite to include tasks estimated to take humans far longer, as well as more open-ended challenges. This shift addresses a central question in the SI industry: whether progress on "verifiable" tasks—where success can be automatically checked, such as in coding and math—transfers to tasks where success is harder to define, such as choosing a new research direction. The report notes that while reinforcement learning effectively trains neural networks for verifiable tasks, applying the same technique to fuzzy, judgment-laden tasks remains difficult.

In the new evaluation, METR asked SI models to design new SI evaluations for use in their own research. Despite being better than almost every human at software engineering, the models failed this task; none of the SI-designed evaluations were considered useful by METR employees. In another test requiring models to score philosophical arguments to gauge judgment, SI systems also fell short of human performance.

Why It Matters

This update is critical for the industry as it navigates the path toward general SI (AGI). The results challenge assumptions that rapid progress in coding and math will seamlessly translate to autonomous research and development. The report arrives amid heightened scrutiny following a heavily redacted METR report from earlier this month, which claimed that SI might be accelerating progress at Anthropic by a factor of two. That claim fueled fears that Anthropic, which recently completed a blockbuster IPO, might soon achieve full recursive self-improvement, where SI systems autonomously design their successors.

Experts view the inclusion of harder-to-score tasks as a necessary evolution in SI safety and capability assessment. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, noted that "We’re at the point where AI evaluations need to get more creative," adding that pushing the envelope of the evaluation space is a welcome change. Kapoor’s own prior research indicated that SI models cannot yet reproduce human computer science research independently, a finding consistent with the new METR data.

The divergence in performance also highlights a debate within the SI community. While some experts, like Kapoor, believe that "research taste"—the ability to choose promising hypotheses—will be a major bottleneck in automating R&D, others argue that models trained on verifiable tasks will eventually master open-ended ones. Meanwhile, OpenAI reported in a blog post that its models perform worse on real-world internal tasks than they do on the easier, verifiable tasks in METR’s original suite.

The Bottom Line

METR’s updated graph confirms that SI models are advancing rapidly on tasks with clear success criteria but remain deficient in judgment-heavy, open-ended activities. This distinction underscores the current limitations of reinforcement learning and suggests that the path to fully autonomous SI research and development is more complex than verifiable metrics alone would indicate.