The gap between open and closed AI models in cyber capabilities keeps shrinking
Reported by The Decoder and MLQ.ai2 sources ↓
In Vals AI’s simulated corporate-network benchmark, OpenAI’s GPT-6 Sol completed 80.8% of attack objectives, while open-weight DeepSeek V4.1 Flash completed 72.8% at about $3 per run, compared with roughly $22.
Key points
- DeepSeek completed 21 objectives in Vals AI’s harder Lighthouse test, compared with Daybreak’s 27 of 31
- Vals says DeepSeek made most of its progress after 100 million tokens, and scored zero in its first 25 million
- The benchmark agents operated alone on isolated machines, starting with a public web address and no human help
- The UK AI Security Institute estimated the open-versus-closed cyber capability gap at four to seven months in July
Sources
Details
Vals AI’s CyberOffense benchmark simulates corporate networks and tests agents working independently from an isolated attack machine. The Decoder reports that GPT-6 Sol is part of OpenAI’s restricted Daybreak cyber program, while DeepSeek V4.1 Flash is an open-weight model.
In the harder Lighthouse test, Daybreak took control of one company’s administrator account, obtained passwords and used them to access a second company. It completed 27 of 31 objectives in under three hours; DeepSeek completed 21, at about one-tenth the cost, according to The Decoder.
The Decoder reports that DeepSeek made most of its progress after 100 million tokens, following no completed objectives in its first 25 million. The UK AI Security Institute’s July analysis put the gap between open and closed frontier models at four to seven months, down from six to ten months for most of 2025.
Written from coverage as of
Images
- The Decoder
- The Decoder
- The Decoder
- The Decoder