No AI Agent Passes More Than 46 of 168 Physics Research Attempts in New FrontierPhysics Benchmark
Reported by Tech Times1 source ↓
FrontierPhysics, a benchmark from UC Berkeley’s Bingran You and 71 co-authors, found that no tested AI agent passed more than 46 of 168 research attempts. Claude Opus 5.5 led with a 27.4% pass rate.
Key points
- Claude Opus 5.5 passed 46 of 168 attempts, the highest result among the agents tested
- All 22 models were tested on 56 research tasks, with three attempts per task
- Every tested agent had a pass rate below 30%
- GPT-6 Astra’s score rose from 25 to 34 passes after nine attempts were corrected
- The benchmark’s authors say failure labels cover only GPT-6 Astra
Details
The benchmark paper was submitted to arXiv on November 4. Its 56 tasks were developed by 42 physicists at 26 institutions from research across nine areas, including condensed matter, atomic physics, astrophysics and fluid dynamics. Eighteen tasks build on work already published by contributors, and four are based on Nature papers.
FrontierPhysics tests a broader research process than solving a bounded physics problem: tasks require literature review, planning, experiments and a written manuscript. Each attempt had a 10-hour limit, and a rollout received no pass unless it cleared the verifier and all binary blockers. The authors describe passing as clearing the task rubrics and an automated reviewer; the paper does not define it as passing journal peer review.
OpenAI’s GPT-6 Astra passed 25 attempts before the authors corrected nine misgraded attempts, bringing its result to 34 of 168, or 20.2%. The paper’s failure analysis covers GPT-6 Astra alone; among its 143 failed rollouts, 48% were tagged for non-rigorous reasoning and 34% for missing physics intuition, with categories able to overlap. The authors also note that non-blocker scores involve subjective judgments and that few tasks per domain limit comparisons between fields.
Written from coverage as of
Images
- Tech Times