Value-Gradient Hypothesis of RL for LLMs

In three linesTheoretical study of why critic-free RL methods (PPO, GRPO) improve LLMs. Authors show actor updates are value-gradient-like in expectation, and autodifferentiation through attention produces empirical costates approximating the value signal. Decomposition of RL impact into value-gradient signal and reachable reward headroom.

Summary generated by Claude — human-verified