The Hidden Power of Scaling Factor in LoRA Optimization

In three linesTheoretical and empirical study of the scaling factor α in LoRA. Authors show α dominates optimization far more than learning rate via a Signal-Drift framework. They discover a square-root law linking α to rank and propose LoRA-α to improve performance and simplify hyperparameter tuning.

Summary generated by Claude — human-verified