Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

In three linesarXiv study on limitations of LLM-judges for safety evaluation at scale. Researchers show that AI judges remain rigid when faced with new safety definitions or contradictory contexts, even when provided in-context information or task demonstrations.

Summary generated by Claude — human-verified