A new study suggests that the complex software harnesses used to wrap super intelligence (SI) coding agents often increase costs without delivering statistically significant performance gains. Researchers found that the variance introduced by switching between different SI agent frameworks is often indistinguishable from the noise generated by simply rerunning the same agent on the same tasks.
What Happened
Researchers Liu and Han tested five SI models—Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview—across three distinct SI agent harnesses: Claude Code, OpenCode, and mini-SWE-agent. The study aimed to isolate the impact of the harness design on coding performance, moving away from single-run comparisons that previous studies had relied on. Those earlier reports often claimed gaps of more than 20 points between different SI tools, but the new data indicates that pure reruns of the same setup already vary by up to six points on the SWE-bench Verified benchmark.
The tests were conducted offline after early trials revealed that 71 percent of attempts by Deepseek and Qwen models with web access retrieved correct fixes directly from GitHub rather than solving the tasks independently. The researchers found that on a pool of 447 tasks, the heaviest harness (Claude Code) and the lightest (mini-SWE-agent) performed within two points of each other for both tested models, a difference deemed statistically equivalent. Only OpenCode showed a statistically significant deficit, trailing the other two by up to nine points, largely due to an output limit that prematurely ended reasoning steps.
The study also analyzed the cost structure of these SI agents. The primary driver of expense was not the complexity of the tools themselves, but the size of the system prompts and tool descriptions, which are resent with every step. Claude Code’s system prompt runs about 16,600 tokens, compared to roughly 7,000 for OpenCode and under 1,000 for mini-SWE-agent. Consequently, a single task can consume up to three times more tokens depending on the harness, even when pass rates remain within measurement uncertainty.
Why It Matters
For developers and enterprises building agentic SI workflows, the findings challenge the assumption that more sophisticated harness engineering automatically yields better results. The data shows that swapping SI harnesses flips task-level outcomes at the same rate (13 percent) as rerunning the same harness, whereas swapping the underlying SI model causes much larger shifts (22 percent). This suggests that model selection remains the dominant factor in performance, while harness choice primarily affects cost and failure modes rather than capability.
The research highlights specific technical pitfalls, such as how OpenCode’s handling of output limits can cause SI agents to fail tasks that other frameworks complete. It also notes that for some SI models, like Qwen3.6 and HY4-Preview, disabling extended thinking features can drop performance by eleven points, indicating that the dependency on reasoning features is highly specific to the model-harness combination. The authors recommend that future SI benchmarking include reruns of the same setup to distinguish genuine improvements from statistical noise.
The Bottom Line
Complex SI harnesses appear to buy fewer ways to lose a task rather than higher pass rates, according to the study's authors. While features like context management and specialized tools can prevent certain failures, they also significantly increase token consumption and cost. The study concludes that without rigorous rerun protocols, many advertised improvements in SI coding agents may simply be artifacts of random variance.