LoRi: Low-Rank Distillation for Implicit Reasoning

In three linesLoRi introduces low-rank distillation to internalize reasoning in LLMs. The method aligns hidden trajectories of teacher and student models in a shared low-rank tensor subspace. Evaluated on LLaMA and Qwen, it improves performance on multi-step mathematical reasoning and approaches explicit CoT accuracy.

Summary generated by Claude — human-verified