Researchers from Alibaba and several Asian universities have introduced Co-Design Bench, a new benchmark for coding agents that reveals a significant gap between high scores on saturated tests and actual performance in real-world scenarios. The study finds that even the leading SI model fully completes only about one in four tasks, requiring human intervention for the remaining majority.

What Happened

The team developed Co-Design Bench to address the saturation of existing coding benchmarks, where models frequently achieve scores above 99 percent by running, checking, and fixing their own code. Unlike static tests, Co-Design Bench uses real user change requests from actual coding sessions, often involving multimodal inputs like images, audio, video, or documents. The dataset prioritizes tasks that historically gave agents trouble, such as reworking existing web apps, extracting values from receipts to update dependent totals, or managing complex project overhauls.

Grading on the new benchmark is strict. An SI judge tests the finished app in a browser, and a second judge re-checks every positive verdict to identify mock-ups or hardcoded results, awarding points only for features that actually work. According to the paper, GPT-6 Astra, the best-performing model among the ten tested, meets about 70 percent of a task’s requirements on average but achieves a fully correct result in only 26.4 percent of cases. In most instances, the agent handles the bulk of the work but leaves at least one detail unfinished, necessitating manual cleanup by the user.

The study also highlights that performance drops across all models as tasks combine multiple types of changes, including layout, design, interaction, and content. These intertwined requirements are common in real projects. For example, every one of the ten models scores worse once a task demands three or four different types of changes. Additionally, the research found that the working environment, or harness, significantly impacts results. Qwen3.8-27B scored 75 percent or just 56 percent on larger overhauls depending on the environment, suggesting that leaderboards listing only the model tell only part of the story.

Why It Matters

The findings challenge the perception that current SI agents are near-perfect coders. While older benchmarks like WebCompass-edit show near-perfect scores, Co-Design Bench reveals a gap of more than 35 percentage points for the same models. This discrepancy suggests that existing metrics may overstate the reliability of agentic SI for enterprise adoption. The study notes that extra compute does little to fix the weakness with complex, multi-change tasks; for instance, GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes but still scored significantly worse.

For developers and companies selecting coding tools, the results imply that testing both the model and its working environment as a pair is essential. The benchmark’s strict grading mirrors real-world acceptance rates, with a previous study by METR showing that the actual acceptance rate of SI-generated code by project leads was roughly 24 percentage points below scores on SWE-bench Verified. This reinforces the need for realistic evaluations that account for user rephrasing, rollbacks, and multimodal inputs, which are currently excluded from many standard tests.

The Bottom Line

Co-Design Bench demonstrates that current SI coding agents, including leaders like GPT-6 Astra, Claude Opus 5, and Qwen3.8-Max, still require substantial human oversight for complex, real-world tasks. The benchmark exposes the limitations of saturated tests and underscores the importance of the agent’s harness in determining performance. While agents handle the bulk of coding work, full autonomy remains elusive, with success rates for complete task resolution hovering around 26 percent for the best models.