SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training

In three linesSystematic study of MoE model compression (Qwen3-Next-80A3B → 23A2B) via pruning and distillation at pretraining scale. Pruning outperforms training from scratch, multi-token prediction (MTP) distillation improves performance, and progressive schedules beat one-shot compression.

Summary generated by Claude — human-verified