What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy

In three linesStudy on grokking (delayed transition from memorization to generalization). Authors show weight norm doesn't directly control grokking delay but acts through logit scale. Fixing norm and varying output temperature, they recover 85% of delay by matching logit scale. Effect is loss-dependent (cross-entropy vs MSE). Logit scale and softmax saturation are the proximal variables.

Summary generated by Claude — human-verified