Shortcuts in the Tail: Debiasing via Post-Hoc Spectral Compression of Fine-Tuning Updates

In three linesA post-hoc method reduces fine-tuning bias by truncating the tail of the SVD decomposition of weight updates (ΔW). Tested on 3 models (0.5B–7B) and 4 benchmarks, it decreases performance gaps on underrepresented groups up to 5× (CivilComments) with <2pp accuracy loss, requiring no retraining or group labels.

Summary generated by Claude — human-verified