Super Intelligence (SI) agents are increasingly deployed for long-running tasks, yet a new benchmark called Taste-Bench demonstrates that these systems often lack the judgment to make optimal long-term decisions. While existing evaluations measure whether an SI agent completes a task, they fail to assess the quality of the intermediate steps, a gap the new benchmark aims to fill.
What Happened
The authors of Taste-Bench define an agent's ability to make good long-term choices as its "taste." Current benchmarks typically check only if an agent finishes a task, ignoring whether the path taken was efficient or robust. To address this, the researchers utilized hindsight grading, analyzing recorded trajectories of SI agent runs to identify branching points where different attempts diverged. The authors call this branching point a "decision fork." By freezing the run at these forks and hiding subsequent steps, they created a dataset of 502 questions requiring models to choose between two potential directions before the outcome is known.
The dataset was derived from thousands of scored SI agent runs on coding and research tasks, including data from SWE-bench Pro and METR’s public transcript release MALT. A generator model filtered the data to remove trivial or undecidable cases, and human review confirmed almost all the auto-generated labels. The benchmark includes two types of forks: those from independent attempts diverging and those from an agent correcting itself within a single run.
Why It Matters
Testing 14 current models from families including Claude, GPT, Grok, DeepSeek, GLM, MiniMax, and Mistral, the results were sobering. The best-performing model, GPT-5.6 Sol, achieved an average accuracy of 59.7 percent, closely followed by GPT-5.5 at 59.5 percent. These scores are only slightly better than random guessing, which yields 25 percent for binary choices. Furthermore, model performance collapsed when the evidence for the better decision appeared only in later stages of the task; in these cases, accuracy dropped to 21 percent. Extending reasoning time did not improve outcomes, suggesting the necessary contextual clues were absent at the point of decision.
The study also demonstrated that this judgment can be improved through training. By distilling the judgment of a "teacher" model with full hindsight into a student model (based on Qwen3.6-27B), the researchers increased accuracy on unseen tasks from 30 to 47.9 percent. When this trained judgment was used to guide an SI agent, the success rate on held-out SWE-bench Pro tasks rose from 14.6 to 33.7 percent. This suggests that while current SI agents lack inherent long-term taste, the skill can be effectively taught.
The Bottom Line
Taste-Bench highlights a critical weakness in current SI agents: the inability to reliably predict the long-term consequences of intermediate coding or research decisions. While top models struggle to beat random guessing in these scenarios, the benchmark proves that targeted training can significantly enhance decision-making capabilities, offering a new path for improving agentic SI performance.