AI is getting better faster than experts can reliably test whether new systems are safe, METR warns
Reported by The Decoder1 source ↓
METR says AI models are improving quickly on research tasks, while gaps in available data and limits of current tests make their capabilities and risks harder to assess.
Key points
- On defined coding and computational tasks with direct feedback, models beat experienced professionals most of the time
- The task length models can handle doubles every 3.6 months, according to METR
- A simple projection of METR's data puts models at a 50% success rate on 200-hour tasks by July 2027
- METR says models still struggle with open-ended tasks requiring judgment and with situations that are hard to reproduce for training
Details
METR says available public data is insufficient to rule out a sharp acceleration in AI research over the next few years. The organization warns this could make it harder for people to understand and respond to catastrophic risks from new AI systems.
On defined coding and computational tasks with direct feedback, models outperform experienced professionals most of the time, but have not surpassed large groups of top experts, METR found. For tasks that take people longer, the task length models can handle doubles every 3.6 months. A straightforward projection of the data suggests a 50% success rate on 200-hour tasks by July 2027; this is an extrapolation, not a reported test result.
METR says models have difficulty with open-ended questions that lack immediate feedback and require judgment, and with situations that are difficult to recreate for training. It also warns that easily run tests may overstate real-world performance because developers train models in environments similar to those tests.
Written from coverage as of
Images
- The Decoder