Generating Pretraining Tokens from Organic Data for Data-Bound Scaling

In three linesSynPro, a synthetic data generation framework, helps LLMs learn more thoroughly from limited organic corpora through rephrasing and reformatting operations. Optimized via reinforcement learning, it unlocks 3.7-5.2x more effective tokens than simple repetition on 400M and 1.1B models, even surpassing the non-data-bound oracle at 1.1B scale.

Summary generated by Claude — human-verified