PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometry

In three linesPermDoRA tests whether adapter interference stems from geometric parameter overlap. On LLaMA-3.1-8B and Mistral-7B, Riemannian-inspired geometry-aware merging provides no advantage over standard averaging in multi-domain settings. Orthogonality of adapter updates weakly predicts composition performance; interference appears driven by shared nonlinear representations instead.

Summary generated by Claude — human-verified