Small LLMs: Pruning vs. Training from Scratch

In three linesComparative study of pruning vs. training from scratch on Llama-3.1-8B (ratios 0.5–0.8, 6 methods). Pruning outperforms random initialization with equal token budget, but advantage narrows with more tokens. Fine-grained pruning retains benefit even with unlimited budget; coarse structured pruning can be matched by training from scratch.

Summary generated by Claude — human-verified