Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs

In three linesA study reveals that in zeroth-order (ZO) optimization for LLM fine-tuning, a single decoding layer dominates adaptation. Fine-tuning this dominant layer alone matches or exceeds full-model ZO fine-tuning on LLaMA2-7B and Qwen3-8B, with speedup up to 4.52×. The dominant layer is identifiable before training via activation-outlier analysis.

Summary generated by Claude — human-verified