A2RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation

In three linesA2RBench is an automated pipeline for generating formally verifiable abstract reasoning benchmarks. Using programmatic verification (cycle consistency), it eliminates hallucinations and scales task variations. Evaluations show current LLMs score 39.8% vs 68.5% for humans, and struggle with complex 3D tasks.

Summary generated by Claude — human-verified