Exploring Starts Are Not Enough: Counterexamples and a Fix for Monte Carlo Exploring Starts

In three linesStudy of convergence properties of Monte Carlo Exploring Starts (MCES) in tabular reinforcement learning. Authors construct counterexamples showing MCES can converge to suboptimal solutions despite initial exploration. A modification scaling learning rates inversely to update frequencies guarantees convergence to optimality.

Summary generated by Claude — human-verified