Dynamic Rollout Editing for Reducing Overthinking in RL-Trained Reasoning Models

In three linesRL-trained reasoning models often generate unnecessary reasoning after finding the correct answer (overthinking). This paper introduces Dynamic Rollout Editing (DRE), a training-time intervention during GRPO that edits successful trajectories continuing after answer emergence, preserving the verified prefix and weakening preference signals for unnecessary thinking.

Summary generated by Claude — human-verified