DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

In three linesDevBench is a telemetry-driven benchmark evaluating LLMs on 1,800 realistic code completion tasks across 6 programming languages. 9 SOTA models tested, best score 43.5% Pass@1. Combines functional correctness, similarity metrics, and LLM-judge assessments on usefulness and contextual relevance.

Summary generated by Claude — human-verified