Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

In three linesPRECISE extends Prediction-Powered Inference for ranking evaluation by combining small human-labeled sets with large LLM-judged sets using Claude 3 Sonnet. Reduces Precision@4 standard error from 4.45 to 3.50 (−21% relative). In production, correctly identifies best system variant from 100 human labels; A/B testing confirms +407 bps daily sales lift.

Summary generated by Claude — human-verified