RegMix-D: Dynamic Data Mixing via Proxy Training Trajectories

In three linesRegMix-D extends RegMix by leveraging full loss trajectories from proxy runs, not just endpoint losses, to predict optimal data mixtures at multiple training stages. Tested on 25B tokens of Pile with a 1B model, RegMix-D outperforms RegMix and DoReMi across 13 downstream tasks while using 75% less proxy compute.

Summary generated by Claude — human-verified