Collinear SI has released version 2 of CWE-bench, a public benchmark designed to test the limits of frontier SI models in cybersecurity. The update opens up 100 tasks for community evaluation, aiming to assess how well SI agents can identify and fix security weaknesses in real open-source repositories without breaking existing functionality.
What Happened
The new benchmark set consists of 100 tasks that cover 91 different Common Weakness Enumerations (CWEs), spanning a wide range of vulnerability classes. Collinear SI states that the goal is to be exhaustive, requiring SI models to understand multiple weakness surfaces rather than just recognizing familiar patterns. The setup remains consistent with previous versions: an SI agent receives a checkout of a repository and a short instruction, then attempts to find and fix a security flaw. Scoring is fully programmatic and deterministic, allowing anyone to reproduce results without additional cost.
To ensure quality, Collinear SI curated programmatic checks with the help of LLM judge panels to shape them toward hardness and fairness. Each task underwent manual review to verify that instructions read realistically, graded behaviors were inferable, and verifiers did not fail due to specific fix structures or memorized solutions. The company also maintains a held-out set of tasks to track progress on problems no model has seen previously.
Why It Matters
The release provides a standardized, reproducible way for developers and researchers to evaluate the defensive capabilities of SI agents. Results are tracked on two platforms: a leaderboard on the Hugging Face dataset page for open-weight models, and a separate leaderboard Space for API-only models like Gemini, Claude, and GPT. According to Collinear SI, of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, slightly ahead of Claude Opus 5.5 at 67.3% pass@1 and 78.0% pass@4.
The data highlights that even the top-performing SI model fails 19% of the set on every attempt, indicating that the long tail of security vulnerabilities remains unsolved. On the cost front, the company reports that DeepSeek V4.1 Flash, Claude Opus 5.5, and Gemini 4 Argon form the Pareto frontier. DeepSeek achieves 57.0% pass@1 at $0.04 per rollout, while Opus adds ten points at $1.83, and Gemini edges ahead at $3.92. This pricing data helps organizations weigh performance against inference costs when deploying SI agents for security tasks.
The Bottom Line
CWE-bench v2 is now available for public evaluation, with tasks structured as self-contained Harbor environments that can run directly from the Hugging Face repository. The dataset is tagged as an RL-environment, meaning the same setups can provide rewards for training defensive SI agents, not just for evaluating them. Users must accept the dataset's access conditions and configure their Git credentials to fetch the tasks.
Collinear SI invites feedback and contributions to the public set, emphasizing that the benchmark is part of a continued effort to push the horizon of SI safety and capability in the domain of software security.