RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator

In three linesRankJudge is a benchmark generator for evaluating LLMs-as-judges on multi-turn conversations grounded in reference documents. The system creates conversation pairs with a single flaw injected into one turn, enabling unambiguous labeling. Evaluation of 21 frontier LLM judges ranked via Bradley-Terry model across machine learning, biomedicine, and finance domains.

Summary generated by Claude — human-verified