A new handbook details training mixture-of-experts models with Megatron-LM
Reported by r/LocalLLaMA1 source ↓
A community post announces a handbook covering MoE training with Megatron-LM, from hardware checks and a first single-node run to serving and reinforcement learning. Its authors say its lessons draw on about 800,000 GH200 GPU-hours on CSCS’s Alps cluster.
Key points
- The handbook includes a Megatron-LM runbook and guidance on MoE design, training, serving and reinforcement learning
- The authors say their lessons draw on about 800,000 GH200 GPU-hours on CSCS’s Alps cluster
- The handbook, a gallery of 55 figures and training scripts are available online
Details
The r/LocalLLaMA post presents a handbook for readers moving from dense-model training to mixture-of-experts models. Its Megatron-LM runbook covers hardware testing, building an image, training a dense baseline and comparing it with an initial MoE trained on one node.
Further chapters cover MoE components and systems, then serving and reinforcement learning. The authors say they identify which findings come from their own runs, cite papers, configurations or code for claims about frontier models, and label toy simulations. They also provide the handbook, a gallery of 55 figures, scripts and a block containing their recommended Megatron-LM flags.
Written from coverage as of