The most misunderstood graph in AI just got a major update
Reported by MIT Technology Review, Superpower Daily and Digg3 sources ↓
METR’s updated time horizon plot adds longer and more open-ended tasks. It finds models lag humans on judgment-heavy work, while some capabilities continue to improve along an exponential trajectory.
Key points
- METR added longer tasks and evaluations that require more open-ended judgment
- No AI-designed evaluation in the test was considered useful for METR’s research
- Models also fell short of humans at scoring philosophical arguments
- Some newer models reached “borderline above the bar” on evaluation design, but were not yet usable
Sources
Details
METR’s updated time horizon plot includes tasks estimated to take humans much longer than those in the original suite, as well as tasks with less clearly defined outcomes. The nonprofit measures models by how long a task is that they can complete, using human completion times to estimate difficulty.
For one new evaluation, models designed tests that METR employees then assessed; none was judged useful for the organization’s research. Models also performed below humans on a task scoring philosophical arguments, though METR employees disagree about how relevant that task is to automating AI research and development.
MIT Technology Review reports that some newer models, including Opus 5.5, were rated “borderline above the bar” on evaluation design, a step up from the clearly below-the-bar ratings of all models a year earlier. The report includes few open-ended tasks because they are difficult to design and score, leaving the pace of future progress on them uncertain.
Written from coverage as of
Images
- MIT Technology Review