Week of June 1, 2026: evaluation in crisis, agents failing the long-horizon test

The dominant theme of this week is not a model announcement but a systemic challenge to how the field measures its own progress. Three papers converge on the same diagnosis: our evaluation metrics are structurally flawed. The geometric study on LLM-as-Judge (arXiv:2606.03043) quantifies what many suspected: across 41 LLM judges and 8 Indic languages, the models' evaluation axis is nearly orthogonal to that of humans (87-89° divergence), and inter-LLM consensus (r≈0.35) systematically exceeds LLM-human alignment (r≈0.27-0.32). Simultaneously, the audit of FOLIO and MALLS reveals 39% and 36% error rates in the FOL formalizations of these reference benchmarks — benchmarks on which dozens of papers have been published. The practical consequence is severe: model rankings on these corpora are partly fictitious, and the +9 to +22 point gains observed after correction on Gemma 31B, Qwen3-30B, and GPT-4o-mini do not reflect model improvement but corrected test data. The clinical counterfactual evaluation (CSS metric) drives the point home: six frontier models ranked similarly on traditional metrics completely invert their ordering when assessed on the ability to adapt oncological recommendations to case mutations, with a universal blind spot on surgical status changes.

The second structural theme is the documented failure of agents on long-horizon tasks. LongDS-Bench (68 tasks, 2,225 turns drawn from real Kaggle notebooks) caps the best models at 48.45% accuracy, with a 47-point drop between early and late turns — long-horizon errors account for 52 to 69% of total failures. DeskCraft confirms the pattern on professional GUI workflows: GPT-5.4, the best agent tested across 538 tasks in design, video, audio, and 3D, reaches only 31.6% in standard mode. MedCUA-Bench adds a critical dimension: in authentic clinical interfaces (OpenEMR), the best closed models achieve 54.2% strict success, while open-source agents average 2.5%. This triptych draws a consistent boundary: current agents handle short, well-defined tasks adequately but lose analytical and procedural coherence once the horizon exceeds a few dozen steps. The Eywa memory architecture (90.19% on LoCoMo, 88.2% on LongMemEval-S) offers a serious direction — immutable source storage, typed validation, deterministic retrieval without LLM calls — but remains a partial solution to a deeper architectural problem.

A third signal, quieter but with strong operational impact, sees two foundational vulnerabilities surface simultaneously. WASH demonstrates that averaging the probability distributions of 3 to 5 models drops z-scores for six major watermarking schemes from 5-300 to below 2 (detection threshold: 4), rendering statistical traceability of generated content practically inoperative. On the internal safety side, the study on linear representations of synthetic deception (Pythia-1.4B, Gemma-2, Qwen2.5-7B, Llama-3.1-8B) shows that linear probes detect deception with AUC ≥0.99 as early as layers 1-3, opening a concrete path for activation-based monitoring — but also confirming that the capacity for coherent deception is encoded very early in the network. On the infrastructure front, the merge of the tensor-mode multi-GPU KV cache fix in llama.cpp b9455 (JohannesGaessler) is the kind of silent fix that unblocks local deployment configurations that have been stalled for weeks. Finally, LEAP solving all 12 Putnam 2025 problems in Lean and reaching 70% on Lean-IMO-Bench (versus <10% for generic LLMs) confirms that formal verification via iterative agentic decomposition is now a mature research direction, not an experimental one.

The coming week will likely see at least one major paper attempt to propose an alternative evaluation protocol to LLM-as-Judge, as the critical pressure has reached a threshold that benchmark teams can no longer easily ignore.