Three Leading AI Models Agreed on Just One of 121 Potential Code Flaws
Reported by Business Insider and Infosecurity Magazine2 sources ↓
Echo found that three AI models flagged just one of 121 potential code issues in common during audits of three open-source packages. The models produced 446 initial findings, of which 42 passed automated proof-of-concept testing.
Key points
- The models shared just one finding among 121 potential issues
- Claude Mythos had 25 validation passes, compared with nine for GLM-5.3 and eight for GPT-5.6 Sol
- The hunts took about 24 minutes for GPT-5.6 Sol, 174 minutes for GLM-5.3 and 198 minutes for Claude Mythos
- Echo said each model ran once and the comparison was not a comprehensive benchmark
- Echo withheld project names and technical details until maintainers are notified
Sources
Details
Echo tested Claude Mythos 5.1, OpenAI's GPT-5.6 Sol and Z.ai's GLM-5.3 on two Maven libraries and one npm package. The models used Echo's hunting framework and the same validation process; Echo accessed GPT-5.6 Sol through OpenAI's Daybreak program for vetted security teams.
The nine runs yielded 446 initial findings, reduced to 152 after cleanup. After duplicates were set aside, 55 findings underwent automated proof-of-concept testing and 42 passed. Echo cautioned that these results were not necessarily 42 distinct bugs, because findings reported by multiple models could be counted for each.
The models differed in their ratings of a flaw in an npm package with more than 350 million monthly downloads: GLM-5.3 rated it Critical, Claude Mythos rated it High, and GPT-5.6 Sol treated it as documented behavior rather than a bug. Automated validation rated it Low. Echo said findings rated Medium or Low were not tested, and automated validation did not confirm bugs with project maintainers.
Echo described the exercise as a controlled comparison, not a comprehensive benchmark, and noted that each model ran only once. It said it would not identify the projects or publish technical details until maintainers had been notified.
Written from coverage as of