Memorization Dynamics of Fill-in-the-Middle Pretraining
Study of verbatim memorization during Fill-in-the-Middle (FIM) pretraining on Llama 3.2. FIM recovers more short or partial spans compared to standard LTR, with extraction growing linearly with repetitions. Suffix context is insufficient: memorization remains anchored in prefix context.