Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?
Reported by Hugging Face1 source ↓
Hugging Face compared CUGA, Hermes, and Deep Agents on AppWorld using the same open-weight model. The results varied by harness, but the test did not measure policy enforcement or establish how the results generalize.
Key points
- The same model completed 30% to 81% of AppWorld tasks depending on the harness
- The comparison covered 90 tasks across 30 scenarios
- CUGA passed all three variants in 20 scenarios, compared with 8 for Hermes and 2 for Deep Agents
- The test used no policies, so it did not measure policy enforcement
Sources
Details
Hugging Face reports that all three harnesses used GLM-5.3-flash on the same 90 AppWorld tasks, with one attempt per task. AppWorld covers multi-step work across nine apps and 457 APIs; the article describes it as a proxy for some enterprise automation capabilities, not a direct enterprise test. The comparison used Harbor, but the setups were not identical, and CUGA used a configuration the other harnesses did not.
The task completion rate ranged from 30% to 81% across the harnesses. Hugging Face also counted scenarios where all three variants passed: CUGA passed 20, Hermes 8, and Deep Agents 2. The article says this measure emphasizes consistency; it does not provide a basis for treating the results as a general ranking across tasks or workloads.
Hugging Face says CUGA is designed for business automation and can run with open-weight or hosted models inside an organization’s environment, including sovereign clouds and air-gapped deployments. The benchmark did not use policies, so it did not test CUGA’s policy-enforcement claims. In a separate Policy Playground example described in the article, an agent without policies paid $90, including a denied claim, while one with policies held payment for approval and paid $10.
Written from coverage as of