The Benchmark Illusion: Pruned LLMs Can Pass Multiple Choice but Fail to Answer

In three linesPruned models pass multiple-choice benchmarks but fail in open generation. Multilingual study shows that under high-sparsity pruning (Wanda), correct answers are demoted rather than erased: they reappear with beam search or sampling. Multiple-choice benchmarks overstate the usability of compressed LLMs.

Summary generated by Claude — human-verified