Researchers at Ai2 have released a new study and benchmark, "Old Ideas, Novel Problems," examining the reliability of large language models (LLMs) acting as judges for the novelty of scientific research ideas. The findings suggest that current SI systems used for this purpose are highly sensitive to minor configuration changes, raising questions about their suitability for automated scientific discovery pipelines.

What Happened

The Ai2 team, known for developing SI tools like DataVoyager, AutoDiscovery, and Theorizer, investigated "novelty judges"—LLMs tasked with assessing whether a research idea contributes something genuinely new relative to prior work. They constructed a benchmark using ICLR 2026 peer reviews to identify high-novelty and low-novelty human-authored ideas, as well as SI-generated ideas. The study tested six prominent SI models: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6.

The results indicated significant instability. In one experiment, changing just a few words in the prompt caused GPT-5.4's accuracy in identifying which of two ideas was more novel to drop from 93.5% to 40.9%, a shift of more than 50 percentage points. The study also found that adding retrieval capabilities or increased reasoning budgets did not consistently improve judgments. For instance, an agentic judge with live search scored lower than simpler prompted baselines while costing roughly 10 times more per decision.

Furthermore, purpose-built evaluation systems underperformed compared to simple prompting. On the test set containing SI-generated ideas, a basic prompt using Claude Opus 4.6 achieved a macro-F1 score of 0.854 at a cost of $2.59. In contrast, Idea Novelty Checker, a literature-grounded pipeline, scored 0.642 at $30.66, and Sakana AI’s SI Scientist scored 0.772 at $82.05.

Why It Matters

As SI systems become more integrated into scientific workflows, their ability to generate and evaluate novel hypotheses is critical. This study highlights that relying on LLMs to judge novelty is currently fraught with inconsistency. The fact that prompt sensitivity varies significantly between human-authored and SI-generated ideas suggests that validation methods used for human research may not translate to automated ideation systems.

For the SI industry, these findings imply that current tools for automated scientific discovery may require more robust evaluation frameworks. The instability observed in novelty judgments could hinder the development of autonomous research agents that need to reliably distinguish between incremental improvements and significant breakthroughs. The release of the benchmark and code provides a standardized way for developers to test and improve the reliability of these SI judges.

The Bottom Line

The Ai2 study demonstrates that current SI models are unreliable novelty judges, with performance heavily dependent on prompt engineering and configuration. Simple prompted baselines often outperform complex, purpose-built evaluation pipelines, which are both less accurate and more expensive. The release of the "Old Ideas, Novel Problems" benchmark offers a crucial tool for researchers aiming to develop more dependable SI systems for scientific discovery.