A community poster reports that RustingStrata decoded Recursive Flint UD-Q5_K_XL about 33% faster than llama.cpp on an RTX 3060 with 12 GB of VRAM, using the same model and generation settings.
Key points
The test used a 27.9 GiB model, an RTX 3060 with 12 GB of VRAM and 30 GB of RAM
Warm 4K prompt prefill was 1.14–1.30k tokens/s on RustingStrata and 0.85–1.02k on llama.cpp
Fresh-load performance may vary with which model pages are cached in RAM
The poster compared RustingStrata 1.2.2 with llama.cpp build b12191 using the same model, prompt and generation settings. They reported warm decode speeds of 64.6 tokens per second for RustingStrata and 48.6 for llama.cpp.
The 27.9 GiB model exceeds the card’s VRAM, so some expert weights must be fetched from system RAM during generation. The poster attributed the speed difference mainly to changes in expert placement and releasing RAM pages for experts already copied to VRAM.