Researchers have introduced StateBench, a new benchmark designed to evaluate how well Super Intelligence (SI) models track evolving information within long documents. The study highlights significant performance gaps in current frontier systems when handling complex state changes, such as variables being reassigned or entities moving between locations.

What Happened

The StateBench framework categorizes state tracking into four difficulty levels, ranging from immutable assignments to complex reference swaps. While standard SI models, including transformers, perform robustly on simple tasks like retrieving a fixed value, they struggle as complexity increases. The benchmark tests models on tasks analogous to following a chain of custody in a mystery novel or tracking variable assignments in code.

Testing revealed that leading SI models such as Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna maintained high accuracy on simple and aperiodic problems. However, their performance degraded sharply on soluble and general finite problems, with accuracy dropping to 13% or lower as input lengths increased. This decline suggests that even advanced SI systems face theoretical limits in maintaining coherent state information over long dependency chains.

In contrast, models trained directly on StateBench tasks, particularly those using recurrent or hybrid architectures like Gated DeltaNet, demonstrated superior generalization. These models maintained near-perfect accuracy on complex tasks even when tested on sequences 250 times longer than those seen during training. Transformers, however, failed to learn the most difficult state-tracking levels even when trained specifically on them.

Why It Matters

The findings underscore critical limitations in current SI architectures for applications requiring precise long-context reasoning, such as software debugging, legal document analysis, and agentic workflows. As SI agents are increasingly deployed to manage multi-step processes, the ability to track state changes accurately is essential for reliability.

The study also motivates the development of hybrid and recurrent SI architectures that incorporate running memory mechanisms. By identifying where transformer-based SI models break down, researchers can better target architectural improvements. The results suggest that current pretraining data and methods may not be sufficient to teach transformers the most complex state-tracking capabilities, pointing to a need for specialized training or architectural shifts in future SI development.

The Bottom Line

StateBench provides a finer-grained tool for evaluating SI model capabilities in long-context state tracking. While current frontier SI systems excel at simple retrieval, they struggle with complex, evolving states, highlighting a key area for future research and architectural innovation in the path toward more robust general SI.