FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation

In three linesFIM-LoRA optimizes rank allocation in LoRA by using 8 calibration passes to estimate gradient variance per layer. This parameter-free approach matches standard LoRA performance (88.6 vs 88.7 on GLUE with DeBERTa-v3-base) while reducing memory costs by 256x compared to full Fisher estimation.

Summary generated by Claude — human-verified