The Expert Strikes Back: Interpreting Mixture-of-Experts Language Models at Expert Level

In three linesComparative study of interpretability in Mixture-of-Experts (MoE) architectures vs dense networks. MoE experts show lower neuronal polysemanticity than dense FFNs, especially with sparse routing. Experts function as fine-grained linguistic task specialists (e.g., closing LaTeX brackets), not broad domain specialists. Code released.

Summary generated by Claude — human-verified