TodayToday / Industry / StoryI trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)ShareWed 28 Oct · 1:43 PM, updated 1:43 PMFirst seen Wed 28 Oct · 1:43 PMUpdated 1:43 PMLimited coverage, taperingIndustryReported by r/LocalLLaMA1 source ↓Sourcesr/LocalLLaMA1:43 PMRead ↗I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)