I removed 85% of gemma's weights (and it is mostly fine)
Reported by r/LocalLLaMA1 source ↓
A r/LocalLLaMA user says they pruned Gemma 4 26B-A4B to 15% of its original size and retained 92% agreement with the original on one classification task. The smaller model is intended for a specific on-device app, not general use.
Key points
- The pruned model agreed with the original on 92% of a classification task
- On the poster’s Mac, it ran at 79 tokens per second using 3.8 GB of memory
- The model is not intended for general tasks; the poster says knowledge-heavy tasks suffer
- Training on a user’s device has not been tested on a real user device
Details
The r/LocalLLaMA poster says they reduced Gemma 4 26B-A4B, a mixture-of-experts model, to 15% of its original size by retaining eight experts per layer. With only those eight experts available, the router no longer selects among experts, and the model becomes dense, with 3.81 billion parameters and a reported 2.75 GB size at Q4_K_M.
On a classification task, the pruned model agreed with the original 92% of the time. The poster says that is close to the 90% agreement between the original 26B model and the 12B model on the same task. On their Mac, the pruned model ran at 79 tokens per second and used 3.8 GB of memory, which the poster describes as roughly twice the speed and half the memory of the 12B model.
The poster says the model was made for an app that runs an on-device language model continuously, where memory use and energy consumption matter. They also say the model performs poorly on tasks requiring knowledge because that knowledge was removed. Their proposed approach trains blocks to reproduce the original model’s computation rather than training directly on answers; the poster says training on a user’s own device and data could use about 5 GB of memory, but has not been tested on a real user device.
Written from coverage as of
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh