Rapid LLM development is outrunning the research meant to explain it, study finds
Reported by The Decoder1 source ↓
Stanford researchers found that many widely cited claims about large language models no longer hold up when tested on newer models, while others have weakened.
Key points
- The study found that 44 percent of tested claims had expired and 34 percent had weakened significantly
- Only 16 percent of the claims still held, while 6 percent had grown stronger
- Scientific citation fabrication varied widely: GPT-6.1 Sol had a 0 percent rate, while Qwen3.7-Flash had 69 percent
- Researchers say findings should be tied to specific models and dates and made available for retesting
Sources
Details
The Decoder reports that Stanford researchers Owen Queen and James Zou tested 50 widely cited claims against 20 models released between July 2024 and September 2026 by eight developers. Of the claims, 44 percent had expired, 34 percent had weakened significantly, 16 percent still held, and 6 percent had grown stronger. Among 22 findings with an estimated expiration date, the average lifespan was 15.7 months.
Claims about model limitations tended to expire faster than claims about preferences or behavior. Five of nine reasoning-related findings had expired, including difficulties counting letters in “strawberry” and retaining information in long contexts. The researchers suggest these errors may be easier to correct because they have a clear target outcome.
In a citation-fabrication test, the researchers used the prompts from a 2023 study in which GPT-3.5 fabricated 55 percent of references. The Decoder reports rates of 0 percent for GPT-6.1 Sol and 1 percent for Claude Sonnet 5.5, compared with 69 percent for Qwen3.7-Flash. Results also varied substantially across models released in the same period: the number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol.
The researchers caution that some weaknesses remain: Claude Sonnet 5.5 failed about 10 percent of two-hop questions even when it answered both component questions correctly. They recommend presenting empirical results as observations about a particular model at a particular time, and documenting model versions, dates, prompts and analysis code so findings can be retested.
Written from coverage as of
Images
- The Decoder
- The Decoder