I ran Qwen3.5-0.8B in pure Assembly
, updated First seen Updated Limited coverage, tapering
Industry
Reported by r/LocalLLaMA1 source ↓
A community post reports running Qwen3.5-0.8B with tokenization, inference and sampling in Assembly, at 6.51 tokens per second on one CPU thread. The 32 generated tokens matched a Transformers reference run.
Key points
- Assembly generated 6.51 tokens per second on one AMD EPYC 7B12 CPU thread
- The Transformers reference reached 4.34 tokens per second
- Both runs produced the same 32 tokens
- The benchmark used 19 input tokens and greedy decoding
Details
The r/LocalLLaMA post says the Assembly implementation runs tokenization, model inference and sampling without Python or libc at runtime. Python is used at build time to prepare tokenizer tables.
In a benchmark with 19 input tokens and greedy decoding, the Assembly run used BF16 weights, while the Transformers reference loaded FP32 weights and used PyTorch fallback kernels. The post reports two runs; it does not describe additional testing.
Written from coverage as of