Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?

In three linesREFLECT is a meta-evaluation benchmark testing LLM judge reliability for supervising deep research agents. Authors define a fine-grained failure taxonomy (process and outcome levels) via controlled interventions on agent execution traces. Finding: best LLM judges achieve <55% accuracy on evidence verification and reasoning failure detection.

Summary generated by Claude — human-verified