RealMath-Eval: Why SOTA Judges Struggle with Real Human Reasoning

In three linesRealMath-Eval, a benchmark of 224 real exam responses, reveals that state-of-the-art LLM judges fail to evaluate authentic human reasoning (MSE ~2.96 vs ~1.17 on synthetic solutions). Analysis shows human errors form a more diverse error space than synthetic errors, with higher information-theoretic surprisal.

Summary generated by Claude — human-verified