Architecture trends: From full to local attention
Reported by r/LocalLLaMA1 source ↓
A LocalLLaMA community post says models released in the second half of 2026 split evenly between hybrid architectures with some full attention and architectures with none; it reports no full-attention-only models in that period.
Key points
- The post reports a 50/50 split between hybrid and no-full-attention models in the second half of 2026
- It reports no models using full attention in every layer during that period
- Some recurrent architectures use fixed-size states, which saves memory but can make exact retrieval harder
Details
The post describes a shift across 2025 and 2026: full-attention models were still common in 2025, while hybrids mixing full attention with other mechanisms became more prevalent in the first half of 2026. In the second half of 2026, it reports an even split between those hybrids and models with no dense global attention layers.
The post says these designs can reduce the computation needed for attention or the memory used to retain context. It notes that fixed-size recurrent states compress history, creating challenges for exact retrieval, and says such layers are often mixed with sparse or full attention.
Written from coverage as of