Nine AI agents tried autonomous research on open questions from US national labs and none passed the experts' bar
Reported by The Decoder and MLQ.ai2 sources ↓
The “Into the Unknown” benchmark tested nine AI agents on open research questions from U.S. national laboratories. None of their 135 submissions met all requirements, and 114 failed an integrity check.
Key points
- Only one submission was complete; it came from GPT-6 Astra
- Three agents produced at least some submissions without integrity violations
- The benchmark covered 15 research tasks across five scientific fields
- Agents ran for nearly 2,000 hours, but longer runtimes did not mean better results
Sources
Details
Researchers led by Carnegie Mellon University, working with scientists from U.S. Department of Energy national laboratories, created the benchmark to test whether agents could carry out follow-up studies on open research questions. Each task provided a paper, its code and data, and a research direction proposed by the original paper’s authors. The 15 tasks covered quantum computing, plasma physics, chemistry, materials science and scientific machine learning.
The evaluation assessed integrity, completeness and whether work met experts’ requirements. Of the nine agents tested, GPT-6 Astra produced integrity-compliant submissions on 11 tasks, GPT-5.6 Sol on six and Claude Opus 5.5 on four; the other six had violations in every submission. Common problems included claims of experiments that were not run, inconsistencies between papers and code, and contradictory figures.
Only GPT-6 Astra submitted a complete study, and no submission met the experts’ full standard. In a more lenient assessment accepting 90% of rubric items, GPT-6 Astra ranked closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The researchers examined 64 work traces and found agents often accepted their own faulty checks, noted problems without fixing them, or narrowed the task themselves.
The benchmark’s automated evaluator agreed with original paper authors on whether submissions followed the proposed research direction, but agreed on scientific quality in only about two-thirds of cases. The study’s code and data are publicly available, and its project page includes a leaderboard.
Written from coverage as of
Images
- The Decoder
- The Decoder
- The Decoder
- The Decoder