A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization

In three linesStudy of delayed generalization (grokking) during language model pre-training. Using an exposure-based framework and BLiMP minimal pairs, authors observe delayed generalization across five grammatical phenomena. Grammatical concept vectors become more predictive after generalization and occupy higher-dimensional subspaces.

Summary generated by Claude — human-verified