[Project] Gemma.c - Gemma 1/2/3 + multimodal inference in one C file (CPU only)
Reported by r/LocalLLaMA1 source ↓
A community developer introduced Gemma.c, a CPU-only inference engine for Gemma 1, 2 and 3, including multimodal models, implemented in a single C file. The developer reports performance benchmarks on a Core Ultra 5 225H.
Key points
- The developer reports Gemma 3 1B INT8 decoding at about 72 tokens per second on a 10-thread setup
- The developer says INT8 decoding is about 3.5 times faster than FP32 and twice as fast as FP16
- Pre-exported INT8 models are available on Hugging Face
- Batching is not supported, and W8A8 does not work for Gemma 1
Sources
Details
The developer's post says the engine supports FP32, FP16, BF16 and INT8 weights, and has no dependencies beyond libc, with OpenMP optional. Image prompts use an embedded image decoder; a separate prompt format supports Gemma 3's pan-and-scan handling of very wide or tall images.
In benchmarks on a Core Ultra 5 225H using WSL2, GCC and 10 threads, the developer reports about 570 tokens per second for Gemma 3 1B INT8 prefill at 10,000 tokens, rising to about 850 tokens per second at 1,000 tokens. The post also reports about 20 tokens per second decoding for the 4B vision model, and notes that BF16 is slow without AVX512-BF16 and native Windows performance is about 1.7 times slower than WSL.
Written from coverage as of
Stocks in this story
Change since open: 20 Nov to 21 Nov 2026
Last refresh