StateBench: Testing how language models track state changes over long documents
Reported by Hugging Face, Allen Institute for Artificial Intelligence and Unite.AI3 sources ↓
Hugging Face introduced StateBench, a benchmark for measuring how language models track changing information across long inputs. Its tests found that frontier models’ accuracy fell to 13% or lower on the longest, most complex tasks.
Key points
- StateBench tests four levels of state-tracking difficulty using comparable sequences of variable assignments
- Frontier models’ accuracy dropped to 13% or lower on the longest general finite tasks
- Hugging Face tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna
- Models performed well on simpler tasks but lost accuracy on harder tasks as inputs grew
Sources
- Hugging FaceRead ↗StateBench: Testing how language models track state changes over long documents
- Allen Institute for Artificial IntelligenceRead ↗StateBench: Testing how language models track state changes over long documents | Ai2
- Unite.AIRead ↗Ai2 Releases StateBench to Measure Long-Context State Tracking
Details
Hugging Face says StateBench is designed to compare state-tracking performance across four levels of complexity. The benchmark uses sequences of variable assignments, with the levels differing in the types of assignments allowed, and presents the same underlying problems in multiple formats.
Hugging Face tested four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The models performed well on simpler tasks, but their accuracy fell as inputs grew on soluble and general finite tasks, reaching 13% or lower on the longest general finite problems.
The lab also evaluated open pretrained models, including 7-billion-parameter Olmo 3 and Olmo Hybrid models. They were pretrained on the same data and had not seen contexts longer than 64,000 tokens during training; most were tested on longer inputs. In separate experiments, 275-million-parameter models trained directly on StateBench sequences of up to 256 assignments were tested on sequences up to 64,000 assignments.
Written from coverage as of