New open-source benchmark reveals reasoning as the biggest bottleneck for AI in cloud operations
Reported by The Decoder1 source ↓
CloudOpsBench tests AI agents on common cloud DevOps tasks. Its creators say failures mainly stem from agents misjudging task scope, missing relevant services or stopping too early.
Key points
- The benchmark covers cloud service fixes, resource removal, access controls, updates and project decommissioning
- The University of Michigan team says reasoning errors account for about 80% of failures
- Tasks run in real AWS and an emulator; extensions for GCP, Azure and OCI are planned
- The benchmark is available on GitHub under an Apache 2.0 license
Details
The Decoder reports that CloudOpsBench version 1.0 contains 45 tasks on AWS and was built by experienced DevOps engineers. The benchmark runs each task both in real AWS and in an emulator. The project also provides a detailed failure analysis.
In results reported by The Decoder, Claude Fable 5.1 completed 60% of tasks and GPT-6 Astra completed 53%. The team led by University of Michigan researcher Ang Chen says about 80% of failures involved agents misjudging task scope, not checking all relevant services or stopping too early; roughly 20% involved incorrect actions, some with irreversible infrastructure consequences.
Written from coverage as of
Images
- The Decoder
- The Decoder
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh