llm-psicotest: a seeded 38-turn conversation to check whether a quant swap changed behaviour
Reported by r/LocalLLaMA1 source ↓
A community developer introduced llm-psicotest, a tool that runs a reproducible conversation against an OpenAI-compatible endpoint and scores responses against a computed key.
Key points
- The tool tests model behavior across capability tasks and protocol turns
- Seeded prompts make runs reproducible across machines
- It is written in Python, uses the standard library and is licensed under MIT
Details
The r/LocalLLaMA post describes llm-psicotest as a way to compare model behavior across quantization or serving changes. It runs a 38-turn conversation: 29 capability items and additional protocol turns, with code scoring each response against a computed key.
The post says seeded generators produce byte-identical prompts across machines. Tests include constrained prose and checking whether output rules given early in the conversation are still followed 20 turns later.
In the developer’s calibration, a Qwen3.8-Flash-Next GGUF model was tested across four configurations and four seeds. Reported average scores ranged from 25.50 to 28.25 out of 29; on the Strata server, IQ3_S outscored IQ2_XS on all four seeds. These are results from the developer’s setup, not an independent evaluation.
The tool supports Python 3.11 and later and is distributed under the MIT license. The post says community results are accepted with details including server version, context size, KV-cache data type and output budget.
Written from coverage as of