I was curious how small local models handle Polish grammar – a quick 15-task test on 8 GB VRAM
Reported by r/LocalLLaMA1 source ↓
A LocalLLaMA community member shared a 15-task test of small local models running through Ollama on an 8 GB RTX 5050. In the first result, gemma2:9b scored 3/8 on Polish tasks and 7/7 on JSON and code checks.
Key points
- The test covers eight Polish tasks and seven JSON and Python checks
- gemma2:9b at Q4_0 scored 3/8 on Polish and 7/7 on JSON and code
- The author says the test is a small sample, not a model ranking
- More models, including Qwen3, Llama 3.1 and Bielik variants, are planned
Details
The author built the test because they wanted a quick way to check Polish-language performance on their own PC and quantization. It runs through Ollama on an RTX 5050 with 8 GB of VRAM. The eight Polish tasks cover grammar and one historical knowledge question; four tasks check JSON, and three check Python syntax.
The initial result for gemma2:9b at Q4_0 was 3/8 on Polish and 7/7 on the JSON and code checks. One cited error was rendering “22 cats” as “dwudziesty dwa koty” instead of “dwadzieścia dwa koty.” The author says answers are graded with regular expressions, failures are checked by hand, and generated Python is parsed but not executed.
The author describes the results as a quick check rather than a ranking. Each model gets one answer per task at temperature 0, with a 2,000-token context and reasoning disabled; the author notes this may disadvantage Qwen3. Planned tests include other quantizations of gemma2:9b, Qwen3:8b, Llama 3.1:8b, gemma3:12b and two Bielik models.
Written from coverage as of