Coding is solved* (exclusions apply)
Reported by r/LocalLLaMA1 source ↓
A community post introduces CRAP, a Gemma 4 E2B fine-tune that generates runnable x86_64 Linux binaries for programs that print a string and exit. Its creator reports mixed evaluation results and a browser demo.
Key points
- CRAP v1.1 generated working binaries for 26 of 40 novel-string tests, according to its creator
- It scored 10/12 on multi-turn tests and 23/25 on tests with unfamiliar exit codes
- The model is limited to binaries that print a string and exit
- The browser demo loads a 3.6 GiB quantized model and was tested in Chrome on an M4 Pro Mac
Details
CRAP, short for Compiler Replacement for Assembling Programs, is a Gemma 4 E2B fine-tune that emits x86_64 Linux ELF binaries as hexadecimal output. Its creator says it does not use a compiler, assembler or tool calls, and that generated binaries must be downloaded and run outside the browser.
The post reports v1.1 results of 20/23 on a held-out legacy set, 26/40 on novel strings, 10/12 on roughly three-turn tests, 18/24 on doubled-character strings and 23/25 on unfamiliar exit codes. The creator says the novel-string result was 8/40 in v1; the multi-turn result was 0/12.
The browser demo uses wllama on WebGPU to load a Q6_K GGUF model of about 3.6 GiB. The creator reports scores of 109/144 for Q6_K, compared with 114/144 for the full-precision model and 3/144 for Q4_K_M. Chrome on an M4 Pro Mac was tested; Firefox loaded the demo but generated slowly, and no other browser testing was reported.
Written from coverage as of