[Paper] No Second Run: Does RSI Stop at Frontier Pre-training? - 801-30B-A3B-Base Coming Soon
Reported by r/LocalLLaMA1 source ↓
Key points
- The 801-30B-A3B MoE was trained from scratch on 11.8 trillion tokens over three months
- The authors describe breaking pre-training into smaller improvement loops rather than repeating a full run
- The authors report the base model outperformed Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, but trailed Qwen3.5-35B-A3B
Details
The paper says the team applied recursive self-improvement (RSI) to a frontier pre-training run from scratch. It argues that a full run cannot be repeatedly tested because it takes hundreds of thousands of GPU-hours over months, and instead describes smaller loops aimed at improving parts of the training process, including data pipelines and infrastructure.
The resulting 801-30B-A3B is a 30-billion-parameter mixture-of-experts model. The authors report training it for three months on 11.8 trillion tokens with a newly formed team.
The reported fine-tuning used 6 billion tokens and no reinforcement learning. The authors say the base model outperformed Qwen3-30B-A3B and Nemotron-3.5-Lightning-30B-A3B, while trailing the larger Qwen3.5-35B-A3B.
Written from coverage as of