Your Experts Have Partners. There's HOPE for Keeping Them Together
Reported by Hugging Face1 source ↓
Hugging Face introduced HOPE, a method for pruning experts from mixture-of-experts models that accounts for pairwise interactions. The company reports that at 50% pruning, HOPE had the best average rank among methods tested and improved agentic coding results by up to 6.1% over the next-best method.
Key points
- HOPE selects experts for removal based on pairwise interactions, not just individual importance
- At 50% pruning, Hugging Face reports HOPE had the best average rank among tested methods
- Hugging Face reports gains of up to 6.1% on agentic coding tasks over the next-best pruning method
- The method uses a calibration dataset to estimate interactions and choose experts to remove
Details
Hugging Face says HOPE is designed to reduce the memory used by mixture-of-experts models by removing experts while accounting for how experts work together. The method scores pairs of experts using their co-activation and outputs on a calibration dataset, then uses those scores to select a set for removal.
The company reports that at a 50% pruning rate, HOPE achieved the best average rank among the methods it compared, with gains of up to 6.1% on agentic coding tasks over the next-best method. The excerpt does not identify the evaluation tasks or provide results for other pruning rates.
Hugging Face gives Qwen3.5-122B as an example of the memory cost: it says the model has 256 experts in each of 48 MoE layers, with expert MLPs accounting for about 95% of its roughly 250 GB parameter memory. Only eight experts per layer are used for a given token, according to the company.
Written from coverage as of
Images
- Hugging Face
- Hugging Face
- Hugging Face