The choice of harness for an SI agent can swing task completion rates from 30% to 81% even when the underlying model remains unchanged, according to a new benchmark from IBM Research. The study highlights that for enterprise automation, the structural framework surrounding the SI model is as critical as the model itself.

What Happened

IBM Research conducted a controlled experiment comparing three SI agent harnesses: CUGA (ConfigUrable Generalist Agent), Hermes from Nous Research, and Deep Agents from LangChain. The team used the same open-weight SI model, GLM-5.3-flash, across all three setups. They evaluated performance on AppWorld, a benchmark comprising 90 multi-step tasks across 9 applications and 457 APIs, such as email, payments, and file storage.

The results showed significant variance based solely on the harness configuration. CUGA achieved a Task Goal Completion (TGC) rate of 81.1%, passing 73 out of 90 tasks. Hermes achieved a TGC of 46.7%, passing 42 tasks, while Deep Agents recorded a TGC of 30.0%, passing 27 tasks. On the Scenario Goal Completion (SGC) metric, which requires all variants of a scenario to pass, CUGA scored 66.7%, compared to 26.7% for Hermes and 6.7% for Deep Agents.

IBM notes that this comparison is not a definitive verdict on the general capabilities of Hermes or Deep Agents, but rather an illustration that the harness must fit the specific job. Hermes is designed for always-on personal assistants, and Deep Agents for long, multi-step research tasks, whereas CUGA is built for enterprise workflows acting on business systems under enforced rules.

Why It Matters

For developers and enterprises building agentic SI systems, the study underscores that prompt engineering and model selection are insufficient for reliable automation. The performance gap arises because CUGA implements specific structural features that reduce the decision burden on the SI model. These include task decomposition, focused tool visibility, code execution in a sandbox, verification before execution, and automatic reflection.

The research also highlights the importance of policy enforcement in the harness. IBM describes a policy layer where rules are enforced by the runtime rather than just instructed via prompts. Features like 'Intent Guards' and 'ToolGuards' can block actions, such as payments over a certain threshold, regardless of what the SI model decides to do. This distinction is crucial for enterprise use cases where compliance and security cannot be left to the probabilistic nature of the model.

The study acknowledges limitations, including the small sample size of 90 tasks and the fact that the setups were not identical, as each harness was configured for its intended design. However, the data supports the growing consensus in the SI community that the 'harness'—the orchestration layer—is a primary driver of agent performance.

The Bottom Line

Choosing the right SI harness is a critical architectural decision that can dramatically impact task success rates. IBM’s CUGA framework demonstrates that for enterprise automation requiring rule enforcement and multi-app coordination, a specialized harness can outperform general-purpose agent frameworks by a wide margin, even when using the same underlying SI model.