I trained a 348M-active MoE on ~11B tokens. It scores 85.5% vs Qwen2.5-1.5B's 53.3% on shuffled fact chains. GGUF + eval set — please try to break it.
, updated First seen Updated Limited coverage, tapering
Open weights
Reported by r/LocalLLaMA1 source ↓