Hugging Face has announced CWE-bench v2, an updated version of their cybersecurity benchmark designed to test frontier cyber capabilities. The new version includes 100 public tasks that anyone can run and evaluate their models on, with full programmatic scoring for deterministic results.
Key facts
| Fact | Detail | The source says |
|---|---|---|
| Number of tasks | 100 | “CWE-bench v2 opens up a public set of 100 tasks that anyone can run and evaluate their models on.” |
| Leading model | Gemini 4 Argon | “Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4.” |
| Scoring method | Programmatic | “Scoring is fully programmatic, so results are deterministic and anyone can reproduce them without extra cost.” |
What happened
Hugging Face has released CWE-bench v2, an enhanced version of their cybersecurity benchmark designed to test the limits of frontier cyber capabilities. The new version includes 100 public tasks that anyone can run and evaluate their models on. The setup remains the same: an agent receives a checkout of a real open-source repository and a short instruction, tasked with finding and fixing a security weakness without breaking existing functionality. Scoring is fully programmatic, ensuring deterministic results that can be reproduced without extra cost. Each task has been reviewed by hand, and the set is open-sourced to improve quality and transparency. The leaderboard on the CWE-bench dataset page tracks open-weight models, while the CWE-bench leaderboard Space includes API-only models such as Gemini, Claude, and GPT. Of the 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, followed closely by Claude Opus 5.5. The benchmark is designed to be run from the Hugging Face repo using Harbor, a Dockerized environment.
In the Fathom Layer index
- Claude — AI Models & Intelligence
What to weigh
- The benchmark includes 100 public tasks.
- The scoring is fully programmatic and deterministic.
FAQ
How many tasks are in CWE-bench v2?
CWE-bench v2 includes a public set of 100 tasks.
What is the scoring method for CWE-bench v2?
Scoring is fully programmatic, ensuring deterministic results that can be reproduced without extra cost.
Which model leads in the CWE-bench v2 leaderboard?
Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4.
Source: Hugging Face
