Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

In three linesarXiv paper shows Mixture-of-Experts (MoE) models outperform dense architectures under strictly equal resource constraints (identical total parameters, training compute, data budget). Researchers identify an optimal activation rate region consistent across model sizes. Validated on ~200 2B-scale and 50 7B-scale models (50 trillion tokens processed).

Summary generated by Claude — human-verified