Nace AI publishes report detailing training of Drex decision models
Reported by r/LocalLLaMA2 sources ↓
Nace AI Research Lab published a 50-page report describing Drex model architectures, training and evaluation. The report says Drex v1.5 scored 93.4% accuracy on a long-context benchmark.
Key points
- The report covers Drex models ranging from 9B to 35B parameters
- Nace says Drex v1.5 reached 93.4% accuracy on its 32K–128K-token benchmark
- Training used about 1.15 million verified decision records
- The report says most performance gains came from supervised fine-tuning
Sources
- r/LocalLLaMARead ↗[R] First paper on how to train actual SOTA Decision Models ever: full playbook - architecture, datasets, training, evaluation, inference
- r/LocalLLaMARead ↗AMA: How we trained a SOTA Decision model, got top 3 on OpenRouter & top on 5 ArtificalAnalysis: full playbook - arch, datasets, training, evals, inference
Details
The report describes several Drex architectures, including dense causal transformers, diffusion language models and mixture-of-experts models. It also discusses attention designs and decision heads.
Nace’s report says training included full-parameter supervised fine-tuning, consistency training and reinforcement learning from adjusted feedback. The report attributes smaller gains and improved calibration to the latter method; it also reports a 1.37% calibration ECE for Drex v2.0 on about 115,000 held-out decisions.
The report identifies limitations including closed-book knowledge and a lack of same-backbone architecture ablations. According to the community post, the models are available in safetensors and GGUF formats, with llama.cpp and Ollama support.
Written from coverage as of