CWE-bench v2: More coverage, quality, and transparency
Reported by Hugging Face1 source ↓
Collinear AI has released a public version of CWE-bench v2 with 100 tasks spanning 91 vulnerability classes. In the reported evaluation, Gemini 4 Argon led 11 models at 67.8% pass@1.
Key points
- The public benchmark includes 100 tasks across 91 CWEs
- Gemini 4 Argon scored 67.8% pass@1 and 81.0% pass@4 across 11 models
- Claude Opus 5.5 scored 67.3% pass@1 and 78.0% pass@4
- At least one evaluated model solved 95 of the 100 tasks
- The benchmark, grader and harness are open; held-out evaluation requires contacting Collinear AI
Details
Each task gives an AI agent a checkout of a real open-source repository and an instruction to find and fix a security weakness without disrupting existing functionality. The tasks use deterministic, programmatic checks; Collinear AI says each task was also reviewed by hand and the checks were curated with help from language-model judge panels. The company maintains a separate held-out set for tracking performance on unseen tasks.
Collinear AI reports that Gemini 4 Argon narrowly led Claude Opus 5.5 in pass@1, while 95 tasks were solved by at least one of the 11 evaluated models. The leading model failed 19% of tasks on every attempt, according to the company. On cost and pass@1, the company identifies DeepSeek V4.1 Flash at 57.0% and $0.04 per rollout, Opus at 67.3% and $1.83, and Gemini at 67.8% and $3.92 as the Pareto frontier.
The public set is available on Hugging Face, where access is gated by acceptance of the dataset’s conditions. Results for open-weight models are tracked on the dataset page; a separate leaderboard covers API-only models as well. Collinear AI says the tasks can also be run through Harbor and used as training environments for defensive agents.
Written from coverage as of