Theoretical Limits of Language Model Alignment
Reported by Apple Machine Learning Research1 source ↓
Apple researchers derive a theoretical limit for reward gains under a fixed KL-divergence budget and report that best-of-N sampling approaches it on safety and summarization tasks, while PPO and GRPO fall short.
Key points
- The paper derives a closed-form upper limit on reward improvement for a fixed KL budget
- A covariance estimate using base-model samples can predict achievable alignment gains
- Reward hacking worsens with larger reward errors and a weaker KL penalty; reward ensembling reduces it
- On safety and summarization tasks, best-of-N came close to the limit, while PPO and GRPO were substantially below it
Details
The Apple Machine Learning Research authors analyze alignment methods that optimize reward while constraining how far a model’s outputs move from a base model. They derive an expression for the maximum reward gain under a fixed KL-divergence budget, with the limit governed by a Jeffreys divergence term rather than the square-root-of-KL term used in prior analyses.
The authors also express the limit as a covariance under the base model, which they say can be estimated from base-model samples. In the proxy-reward setting, their analysis finds that the difference between ideal and proxy alignment grows with reward error and as the KL penalty decreases; they prove that combining rewards can reduce this effect.
For safety and summarization, the paper reports that best-of-N sampling came close to the derived limit, while PPO and GRPO remained substantially below it. The authors say reaching the limit without high inference costs will require algorithmic improvements.
Written from coverage as of
Images
- Apple Machine Learning Research
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh