When Models Disagree: Rethinking LLM Evaluation for Public Comment Analysis

In three linesResearchers propose a disagreement-based audit pipeline to evaluate LLMs deployed by federal agencies for categorizing public comments. Analyzing 1,260 USDA comments across four LLMs, inter-model thematic divergence exceeds within-model prompt variation, and human annotators introduce interpretive framings absent from the ensemble's collective output.

Summary generated by Claude — human-verified