Evals (evaluations) are standardized tests that measure an AI model's capabilities and limitations across specific tasks. Eleuther AI's lm-evaluation-harness, for instance, is a widely used open-source framework to benchmark language models consistently.
Systematic audit of FOLIO and MALLS benchmarks reveals 39% and 36% errors in FOL formalizations respectively. Authors release corrected annotations and an LLM-based framework to guide manual relabeling, achieving 90% dataset accuracy by reviewing <24% of instances versus >70% for unguided review. Testing on Gemma 31B, Qwen3-30B, and GPT-4o-mini shows +9 to +22 percentage point accuracy gains.
arXiv study showing LLMs struggle to infer user sociodemographics from single conversation histories. Disparities in advice (legal, medical, financial) are minimal but present. Conversation topics prove more predictive than sociodemographic data and affect LLM outputs unpredictably.
arXiv paper on LLM representations: lexical overlap influences embeddings more than semantic content, persists across layers and architectures, and degrades performance in summarization and model editing tasks.
Benchmark evaluating environmental attitudes across 31 LLMs (proprietary and open-weight). Models exhibit more progressive environmental positions than average German survey respondents, but show no systematic relationship with model origin, size, or release context. Detects prompting manipulation risks and sycophantic shifts.
ToolGate is a lightweight controller that decides whether to execute or skip tool calls proposed by vision-language agents. Across five benchmarks with Qwen3-VL, it reduces token cost to 64-69% of the ReAct baseline while preserving accuracy, and further improves accuracy by 1.65 points with matched-domain training.
Study of 'handoff debt': the rediscovery cost when a coding agent resumes an interrupted task. Across 75 tasks and 724 runs, structured notes reduce median agent events by 20–59% and tokens by 42–63% vs. repository-only takeover. Agent benchmarks should evaluate resumption efficiency, not just task resolution.
Study across 6,000 task-condition pairs shows multi-agent debate degrades generation (-1.6 to -15.5pp) via critique-induced confusion, yet improves error detection (+27.4pp F1). Adversarial separation with code-execution grounding and evidence-gated generation achieves +5.3pp on generative tasks.
ChatHealthAI aligns structured EHR representations from a pretrained EHR foundation model with a frozen LLM's semantic space via a task-aware resampler. The multimodal framework integrates longitudinal patient representations with refined clinical event descriptions, improving interpretable clinical reasoning while maintaining competitive predictive performance on the EHRSHOT benchmark.
BehaviorBench is a benchmark for evaluating personalized decision modeling from real-world behavioral traces. Built on 2,000 wallets with 141,445 belief-prediction instances and 1,485,972 trade-prediction instances, it tests whether generative models can adapt predictions to individual users without relying on simulated behavior.
TriEval is an LLM evaluation pipeline assessing bias, toxicity, and truthfulness simultaneously with minimal resources. Compatible with open-source and closed-source models, runs on standard laptop without GPU. Tested on Llama 3 8B, Mistral 7B, Gemma 2 9B, and Claude Haiku, revealing toxicity and truthfulness differences between models.
A r/LocalLLaMA user argues that quantization benchmarks focus on perplexity and prose quality but ignore tool call validity. They hypothesize that quantization errors degrade structured outputs (JSON, schemas) earlier than free text, making current metrics inadequate for agentic use cases.
Starbucks withdraws NomadGo, its computer vision tool deployed across 11,000 stores after nine months. The system, validated at 99% in controlled environment, failed at production scale, highlighting the gap between sandbox testing and real-world deployment.
CVE-Bench evaluates 5 frontier models on 20 real-world CVEs (Pillow, GitPython, urllib3, etc.) across 300 runs. Max solve rate 50% (60% under advisory). Agents patch syntactically but leave vulnerabilities open. Significant cross-family gaps (OpenAI vs Laguna, p<0.05), within-family noise. Failure modes: wrong-search drift, hallucinations, context loss.
Asymptotic theory for SGLD-Gibbs combining stochastic gradient Langevin dynamics with Gibbs updates for Bayesian inference in latent variable models. Authors derive jump-diffusion limits and propose hyperparameter tuning guidance ensuring statistically meaningful uncertainty quantification.
Study on adaptive data selection for wearable prediction across 3 modalities (heart rate, activity, EMA) in longitudinal dataset. AUROC gains up to 0.7 for low-baseline participants, strong inverse correlation (r=-0.67) between improvement and baseline performance. Limited benefit for strong baselines.
Humor generation system for SemEval-2026 Task-1 (MWAHAHA) using a "generate-many → select-best" strategy. Diverse candidate generation via multi-step prompting and model ensembling, then selection using a preference model trained on 2.5K human pairwise judgments. Ranked 1st in English and Chinese subtasks, 2nd in Spanish.
Multi-domain red teaming framework evaluating 11 LLMs across 690 clinical scenarios. Results: substantial variance (scores 0.791–0.984), safety-critical failures masked by aggregate accuracy, 10-20% error amplification on equity tasks. Hybrid evaluation (automated + human validation) essential.
DLLM-JEPA pairs JEPA with masked-diffusion language models for self-supervised representation learning. Eliminates need for explicit multi-view data and reduces training FLOPs by 33% vs LLM-JEPA. Achieves +18.7pp improvement on GSM8K (LLaDA-8B) and +11.4pp (Dream-7B) while preserving base model capabilities.
Study of agreement metrics for LLM-as-Judge evaluation. Analysis of 24 recent papers shows that for binary criteria (MET/UNMET), Pearson r, Spearman ρ, Kendall τ_b, and phi are redundant. Cohen's κ alone adds information. Authors propose a reporting checklist including judgment scale, abstention handling, and confusion matrix.
RealityTest evaluates whether AI systems disclose their identity when asked. Multimodal, multilingual benchmark based on 3,152 identity-probing queries from ~750 participants across 49 countries, 5 languages (text and speech). Findings: only 31% ask directly; a single suppression instruction reduces disclosure below 30% even in best-performing models.
Study of automatic assessment for massive multilingual bitext: decomposed into parallelism evaluation via multilingual embeddings and reference-free quality estimation. Benchmarks 4 embedding models and 9 evaluators on FLORES-200 covering 6,654 language-pair directions. Key finding: no single model is universally reliable; direction-aware routing and calibration required.
New automated metric (Triangulated Preference Shift score) to measure lexical biases introduced during preference learning (RLHF) in LLMs without manual curation. Analysis across 6 model families reveals a shift toward a 'language of prestige' (overuse of 'delve', 'furthermore').
Academic paper proposing product-aware autoencoders for anomaly detection in multi-product cyber-physical systems. Traditional global models create blind spots where attacks can evade detection. Tests on Tennessee Eastman Process benchmark: product-aware model achieves 100% detection accuracy versus 22.2% for global baseline in attack scenarios.
Empirical study adapting Phi Silica for short-form text rewriting via dataset curation, prompt distillation, and parameter-efficient fine-tuning. Dataset built from public slide decks. Results: improved semantic fidelity, reduced hallucinations, higher preference win rate vs GPT-5-chat.
arXiv study on LLM adaptation limits for annotation tasks. Toxicity detection experiments across diverse datasets show 66% of zero-shot errors resist correction via prompting (rescue rate 34.8%). Models follow misaligned definitions while maintaining confidence. Definition-Specific Familiarity (DSF) metric correlates with performance (r=+0.41), outperforming memorization metrics.
Interactive reasoning evaluation benchmark with 474 executable games. LLMs receive only task rules, must query a hidden environment, integrate partial observations, and decide when to submit answers. Evaluates contextual robustness, metacognitive adaptation, and interaction efficiency across frontier models.
Hoeffding Concept Bottleneck Models (HCBM) replace linear concept aggregation with non-linear sparse decomposition based on gradient-boosted trees. Robust to inter-concept leakage, they improve explainability of computer vision predictions, particularly on overhead imagery.
FETCH, an automated legal triage classifier, generates follow-up questions using a low-cost LLM ensemble. The study shows cheap models perform well at classification, but high-quality plain-language question generation requires GPT-4 or higher. Prompt engineering alone is insufficient; LLM-as-judge ratings diverge from human evaluations.
New set-distance reward method for reinforcement learning of vision-language models on chest X-ray report generation. Tested on Qwen3-VL, Gemma3 with GRPO: 6.80% (BERTScore), 7.82% (RadGraph F1), 4.45% (CheXbert F1) improvements over supervised fine-tuning. Enables test-time best-of-N selection and mid-generation pruning reducing tokens by 50%.
Novel approach for Major Depressive Disorder detection from EEG without data augmentation. SGC (Score-Guided Classification) uses an unsupervised generative network to model pathological anomalies as prior, fused with deep feature representations. Cross-Channel Spatial Adaptation module handles multi-center channel heterogeneity. Validated on Mumtaz2016 and MODMA datasets.
PRAXIS is an algorithm that efficiently computes Rashomon sets (sets of near-optimal models) for sparse decision trees. The approach drastically reduces memory and runtime while recovering most of the full set. Code available on GitHub.
Study of delayed generalization (grokking) during language model pre-training. Using an exposure-based framework and BLiMP minimal pairs, authors observe delayed generalization across five grammatical phenomena. Grammatical concept vectors become more predictive after generalization and occupy higher-dimensional subspaces.
Study of temporal framing in news: a taxonomy of 8 temporal frames identifies how references to past, present, and future function as rhetorical devices. Annotated corpus of 458 English/German articles (2K+ sentences, 3K annotations). Supervised models substantially outperform zero-shot approaches for sentence-level temporal framing detection.
FLaG is a lightweight hallucination detection framework for LLMs that models correctness through latent evidence groups. Using energy-based routing and log-marginal aggregation, it captures heterogeneous hallucination patterns without modifying the underlying model. SOTA results across multiple benchmarks with robust transfer across datasets.
Online, distribution-free framework for controlling Conditional Value-at-Risk (CVaR) in non-stationary and adversarial environments. Combines conformal tail risk control, online learning, and Rockafellar-Uryasev variational representation. Provable safety guarantees for nonlinear tail risk under arbitrary data-generating processes. Applications: portfolio risk management and LLM toxicity mitigation.
Perturbative approach for nonparametric instrumental variable (NPIV) estimation inspired by perturbation theory in physics. Extends kernel ridge methods with systematic higher-order corrections. Reduces prediction error by up to 99% in high-dimensional ill-defined cases (β > 0.7) compared to standard ridge regression.
KG-Guard detects hallucinations in knowledge base question answering (KBQA) systems using an augmented graph and lightweight encoder. The model achieves F1 scores of 82.0–87.4 on WebQSP/ComplexWebQuestions with 305× fewer parameters than baselines, and improves downstream KBQA F1 by 13–14.5 points through iterative refinement feedback.
AEyeDE introduces an attention-based attribution framework for detecting AI-generated text using attention matrices from a proxy Transformer model. A lightweight CNN learns discriminative representations from these attribution maps. The method outperforms text-only baselines, shows strong generator-specific detection, and demonstrates robustness under cross-dataset transfer and spelling perturbations.
TrustLDM is a trustworthiness benchmark for Language Diffusion Models (LDMs) covering safety, privacy, and fairness. Results show LDMs degrade alignment when malicious post contexts are attached to masked responses, regardless of context length. An automatic evaluation framework (TrustLDM-Auto) systematically identifies vulnerable configurations across all tested models.
Modern LLMs systematically overestimate their competence and attempt unsolvable queries. Researchers propose Capability Self-Assessment (CSA), formulated as a policy-learning problem using reinforcement learning, to teach models to recognize their limits. RL significantly outperforms supervised fine-tuning, preserves original capabilities, and generalizes out-of-distribution.