Are We Evaluating Knowledge or Phrasing? Mitigating MCQA Sensitivity with ParaEval

In three linesParaEval, an evaluation framework, corrects MCQA benchmark biases by testing models with multiple paraphrases per answer option. Standard scores conflate familiarity with exact phrasing and actual understanding. On 1B-8B models with identical knowledge, ParaEval reduces artificial performance gaps from 2 points to below 1 point.

Summary generated by Claude — human-verified