I'm starting to think a specialized 4B is enough for most of the tasks I'd use an LLM for
Reported by r/LocalLLaMA1 source ↓
A community member reports useful results from training specialized 4B models for 3D scene generation and says an experimental 350M model is beginning to produce recognizable scenes. The tests are ongoing and are not final results.
Key points
- The author reports that specialized 4B models can produce more complex 3D scenes than the 350M model
- The 350M model has produced useful results on some coding tasks up to roughly 16K tokens
- The author reports good 4B results with contexts approaching 96K tokens
- The 350M model is still training, and its release depends on whether its quality improves
Details
The post describes experiments training small models for specific tasks, including generating 3D assets and scenes. The author says the 4B models handle more complicated scenes, while the 350M model has begun completing code and producing recognizable scenes after additional training epochs.
The author reports useful results from the 350M model on medium-complexity coding tasks with contexts of roughly 16K tokens, but says it remains noticeably weaker than the Qwen models. Some outputs fail to render without fixes, results vary by prompt, and long contexts are a problem. The 4B experiments have produced good results at contexts approaching 96K tokens, according to the author.
The post argues that dataset quality and extended training may strongly affect results, noting that evaluation loss did not reliably indicate which checkpoint produced better code. The author also says fully fine-tuning a 350M model can be done on an 8GB VRAM GPU with memory optimizations, though longer contexts and larger datasets increase training time. The models and expanded 3D dataset are not yet released; the author says a 350M release depends on its further progress.
Written from coverage as of