Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource

In three linesarXiv paper demonstrates that Mixture-of-Experts (MoE) models can outperform dense architectures under strictly equal resource constraints (parameters, training compute, data). Researchers identify an optimal activation rate region consistent across model sizes. Validated on ~200 2B-scale and 50 7B-scale models (50 trillion tokens processed).

Summary generated by Claude — human-verified