LLM-Safety Evaluations Lack Robustness

In three linesarXiv paper argues current LLM safety evaluations lack robustness due to small datasets, methodological inconsistencies, and unreliable setups. Systematically analyzes the evaluation pipeline—dataset curation, automated red-teaming, response generation, LLM judges—and proposes guidelines to reduce noise and improve comparability of attack/defense research.

Summary generated by Claude — human-verified