Systematic audit of FOLIO and MALLS benchmarks reveals 39% and 36% errors in FOL formalizations respectively. Authors release corrected annotations and an LLM-based framework to guide manual relabeling, achieving 90% dataset accuracy by reviewing <24% of instances versus >70% for unguided review. Testing on Gemma 31B, Qwen3-30B, and GPT-4o-mini shows +9 to +22 percentage point accuracy gains.
Padyam2Gadyam: dataset of 600 classical Telugu poems (13th-17th centuries) with human-verified modern Telugu and English prose translations. Evaluation of 5 contemporary LLMs shows insufficient performance and limitations of existing MT evaluation metrics for this task.
Researchers propose Bank of Values (BoV), replacing context-dependent value vectors with context-free vectors stored as sparse parameters in the last third of layers. On 135M and 780M models, BoV improves validation loss and performance across 21 benchmarks with reduced compute and memory.
arXiv study showing LLMs struggle to infer user sociodemographics from single conversation histories. Disparities in advice (legal, medical, financial) are minimal but present. Conversation topics prove more predictive than sociodemographic data and affect LLM outputs unpredictably.
arXiv paper on LLM representations: lexical overlap influences embeddings more than semantic content, persists across layers and architectures, and degrades performance in summarization and model editing tasks.
Benchmark evaluating environmental attitudes across 31 LLMs (proprietary and open-weight). Models exhibit more progressive environmental positions than average German survey respondents, but show no systematic relationship with model origin, size, or release context. Detects prompting manipulation risks and sycophantic shifts.
IdiomX is a large-scale multilingual benchmark for idiom understanding, containing 190K+ contextualized examples across 12K+ idioms in English, Arabic, and French. The dataset includes idiomatic/literal usage labels and linguistic metadata. Four tasks evaluate idiom detection, retrieval, and interpretation.
DraDDP is the first public multimodal dataset for discourse parsing in multi-party dialogues. Built from American TV dramas, it contains 495 dialogue segments (6,374 utterances, 9.1 hours of video). Benchmarks demonstrate the value of multimodal information for identifying dependency structures and relation types between utterances.
DOPA, a demonstration retrieval framework, uses an OOD proxy to approximate the inaccessible target domain and guide selection of relevant demonstrations. A Mahalanobis distance-based global diversity constraint ensures sufficient variety among retrieved examples. Positive results across multiple LLMs and tasks under severe distribution shift.
ART (Attention Run-time Termination) is a lightweight runtime mechanism that halts KV block accesses during decoding once their attention contribution becomes negligible. Tested on LongBench, it achieves 20% higher generation throughput on large batches while maintaining comparable accuracy.
CSRP, a three-stage framework for Chinese grammatical error correction, combines continual pre-training (5.9M samples), Chain-of-Thought fine-tuning, and policy optimization with efficiency-aware rewards. Achieves 50.99 F₀.₅ on NACGEC and outperforms GPT-4 on spelling correction (59.61 F1).
SENSE improves retrieval-based speculative decoding by anchoring retrieval on target model hidden states for robust semantic alignment. A soft-gated evaluation module validates semantic equivalence rather than surface forms. On LLaMA and Qwen, SENSE achieves 4.09 mean acceptance length and 3.26x speedup.
Humor generation system for SemEval-2026 Task-1 (MWAHAHA) using a "generate-many → select-best" strategy. Diverse candidate generation via multi-step prompting and model ensembling, then selection using a preference model trained on 2.5K human pairwise judgments. Ranked 1st in English and Chinese subtasks, 2nd in Spanish.
ProactiveLLM learns when to interact with streaming inputs without external signals. Through monotonic random masking and synchronized privileged self-distillation, the model perceives semantic sufficiency from partial inputs. Reduces interaction latency while maintaining quality on text and speech streaming tasks.
Multi-domain red teaming framework evaluating 11 LLMs across 690 clinical scenarios. Results: substantial variance (scores 0.791–0.984), safety-critical failures masked by aggregate accuracy, 10-20% error amplification on equity tasks. Hybrid evaluation (automated + human validation) essential.
DLLM-JEPA pairs JEPA with masked-diffusion language models for self-supervised representation learning. Eliminates need for explicit multi-view data and reduces training FLOPs by 33% vs LLM-JEPA. Achieves +18.7pp improvement on GSM8K (LLaDA-8B) and +11.4pp (Dream-7B) while preserving base model capabilities.
Study of agreement metrics for LLM-as-Judge evaluation. Analysis of 24 recent papers shows that for binary criteria (MET/UNMET), Pearson r, Spearman ρ, Kendall τ_b, and phi are redundant. Cohen's κ alone adds information. Authors propose a reporting checklist including judgment scale, abstention handling, and confusion matrix.
RealityTest evaluates whether AI systems disclose their identity when asked. Multimodal, multilingual benchmark based on 3,152 identity-probing queries from ~750 participants across 49 countries, 5 languages (text and speech). Findings: only 31% ask directly; a single suppression instruction reduces disclosure below 30% even in best-performing models.
DeSQ decomposes complex questions into atomic constraints to generate SPARQL queries. The framework operates in three stages: decomposition, generation of SPARQL fragments with placeholders, and final assembly. Outperforms state-of-the-art on 4/5 major benchmarks with improved robustness to lexical variation.
Study of automatic assessment for massive multilingual bitext: decomposed into parallelism evaluation via multilingual embeddings and reference-free quality estimation. Benchmarks 4 embedding models and 9 evaluators on FLORES-200 covering 6,654 language-pair directions. Key finding: no single model is universally reliable; direction-aware routing and calibration required.
TOPD (Trajectory-aware On-Policy Distillation) improves LLM reasoning by using near-future trajectory information to identify truly divergent states. On AIME24/25, TOPD reaches 63.3%/53.3% vs 60.0%/46.7% in standard OPD, showing 30% of high-loss tokens are false positives.
New automated metric (Triangulated Preference Shift score) to measure lexical biases introduced during preference learning (RLHF) in LLMs without manual curation. Analysis across 6 model families reveals a shift toward a 'language of prestige' (overuse of 'delve', 'furthermore').
ProtStructQA is an executable benchmark for protein structural question answering. 382.2K questions generated from hidden domain-specific language, evaluated on Qwen3 (0.6B–8B) and Gemma-3. Key finding: capability threshold between Qwen3-1.7B and 4B where models transition from failing to produce executable denotations to mastering chain-of-thought reasoning.
Empirical study adapting Phi Silica for short-form text rewriting via dataset curation, prompt distillation, and parameter-efficient fine-tuning. Dataset built from public slide decks. Results: improved semantic fidelity, reduced hallucinations, higher preference win rate vs GPT-5-chat.
arXiv study on LLM adaptation limits for annotation tasks. Toxicity detection experiments across diverse datasets show 66% of zero-shot errors resist correction via prompting (rescue rate 34.8%). Models follow misaligned definitions while maintaining confidence. Definition-Specific Familiarity (DSF) metric correlates with performance (r=+0.41), outperforming memorization metrics.
LaSR proposes a training paradigm for Speech LLMs featuring latent reasoning aligned to acoustic feature regions of target words. Without explicit intermediate tokens, the method improves specialized vocabulary recognition on Fun-Audio-Chat. A new Spoken Darwin-Science corpus for academic terminology is introduced.
SelSkill, a dual-granularity preference-learning framework, optimizes selective skill invocation in agentic tasks. On ALFWorld with Qwen3-8B: +10.9 pp task success, +29.1 pp execution precision. On BFCL: +5.7 pp task success, +29.5 pp execution precision. Zero-shot transfer to Tau-bench and PopQA.
Theoretical and empirical study of parameter-based knowledge editing limits in LLMs. Authors prove via dimensional collapse hypothesis that localized modifications propagate global interference degrading model capabilities. Retrieval-based methods consistently outperform parameter-editing approaches.
SPADER is an RL framework for tool-augmented LLM agents in Multi-Answer QA. It introduces Step-wise Peer Advantage (SPA) for fine-grained credit assignment over long trajectories, and a diversity-aware exploration reward promoting rare entity discovery. Evaluated on QAMPARI, Mintaka, WebQSP, QUEST: improves recall and F1 vs prompting and supervised RL baselines.
Study on preventing catastrophic forgetting during continual pretraining of multilingual language models. Authors propose five parameter alignment strategies (layer freezing, regularization, post-hoc reversion, model merging) tested across 32 languages and four evaluation axes. Parameter alignment substantially reduces forgetting while maintaining language acquisition.
Study of temporal framing in news: a taxonomy of 8 temporal frames identifies how references to past, present, and future function as rhetorical devices. Annotated corpus of 458 English/German articles (2K+ sentences, 3K annotations). Supervised models substantially outperform zero-shot approaches for sentence-level temporal framing detection.
AEyeDE introduces an attention-based attribution framework for detecting AI-generated text using attention matrices from a proxy Transformer model. A lightweight CNN learns discriminative representations from these attribution maps. The method outperforms text-only baselines, shows strong generator-specific detection, and demonstrates robustness under cross-dataset transfer and spelling perturbations.
TrustLDM is a trustworthiness benchmark for Language Diffusion Models (LDMs) covering safety, privacy, and fairness. Results show LDMs degrade alignment when malicious post contexts are attached to masked responses, regardless of context length. An automatic evaluation framework (TrustLDM-Auto) systematically identifies vulnerable configurations across all tested models.
Study combining cognitive-linguistic features (first-person pronouns, absolutist words, negative emotions) with DistilBERT embeddings and Holographic Reduced Representation to detect depression in Reddit posts. Hybrid model achieves F1=0.94 vs 0.80 TF-IDF baseline, AUC=0.981.
TCAR-Gen combines query-conditioned graph neural networks, temporal evidence fusion, and chain-of-trees reasoning for retrieval-augmented generation. Achieves 0.3738 Recall@5 on Victorian Crime Diaries benchmark, outperforming Vanilla RAG, Temporal RAG, and GraphRAG variants. Cross-model evaluation across GPT-OSS 20B to TinyLlama 1.1B shows robust retrieval coverage at smaller scales.
Comparative study of a two-hop Graph-RAG architecture versus standard vector-only RAG for cross-entity financial sentiment analysis. On 100 queries (30 direct, 70 relational), Graph-RAG improves entity recall (+6.4%, p<0.001) and answer relevancy for complex queries (+11.7%), with no quality degradation, modest 22.6% latency increase but 80% variance reduction.
BiGRU architecture enhanced with KAN (Kolmogorov-Arnold Network) block for legal document classification and summarization in low-resource multilingual setup. Evaluation on Bengali/English/transliterated corpus from Bangladesh: 67.96% accuracy in classification (F1=0.65), ROUGE-1/2/L scores of 0.38/0.23/0.31 in summarization. Ablation study shows KAN improves classification from 57.34% to 67.96%.
BOUTEF is a multilingual corpus from 2 countries (Algeria, Tunisia) covering fake news, authentic narratives, comments, and debunking. Includes MSA, Algerian/Tunisian dialects, Arabizi, French, English, and code-switching. Analysis shows fake news relies on emotionally charged narratives and sensational framing, while debunking adopts a factual, verification-oriented style.
Audit of 7 LLMs (US/China) on 2,520 responses to 60 legal-administrative prompts in English and Mandarin. Models default to the institutional framework of input language: 74.5% of English responses adopt US framework, 53.3% of Chinese responses adopt China framework. Risk of jurisdictional misselection when preferred language differs from applicable jurisdiction.
SALSA adapts speech-aware LLMs via layer-wise learned steering vectors optimized with supervised objectives. Tested on children's speech, multilingual, and Mandarin-English code-switching, it achieves up to 46.8% relative improvement over zero-shot. Steering the encoder's later layers outperforms steering the LLM backbone.
UniKE, the first benchmark for cross-modality knowledge editing in unified multimodal models (UMMs), reveals a critical gap: text-side efficacy reaches 92% but VQA accuracy in image generation drops to 18.5%. A reasoning-augmented parameter editing method improves results by up to +18.6 percentage points.
MERIT, a multi-horizon memory retrieval framework for text-to-SQL agents, maintains episode-level and turn-level memory with learned retrieval policies optimized via reinforcement learning. On BIRD-Interact, it outperforms no-memory and static-retrieval baselines, improving success rate while reducing interaction turns.
arXiv paper on autonomous agentic data engineering for model specialization. GPT-5.2 constructs a training curriculum improving a student model by 57.29% through iterative, agent-driven data adaptation. Formalizes a novel task evaluating LLMs as autonomous data engineers.
Activation steering study across four multilingual LLMs (5 figurative categories, 6 languages). Directions learned in one language transfer effectively to others, particularly German. Composite cross-lingual directions match or exceed native directions, providing direct evidence of reusable but target-dependent figurative signals across languages.
CSRM (Configurable Safety Reward Model) jointly optimizes calibrated safety compliance and reward modeling to adapt LLMs to heterogeneous and evolving safety requirements. Achieves 94.6% F1 on CoSApien and 75.8% F1 on DynaBench without additional human annotation.
Comparative study of zero-shot multi-label topic classification using knowledge graphs extracted from documents. Framework tested on 15 LLMs and 8 datasets: keyword-enhanced variant outperforms baseline, graph augmentation helps small models but hurts large ones, and self-consistency decoding increases costs fivefold without performance gains.
Study of global narrative dominance in LLMs via CulturalNB, a dataset of 717 Bengali cultural instances with parallel English-Bangla question-answer pairs. Evaluation of 9 LLMs shows English questions increase global substitution and reduce local perspective coverage, even with local evidence provided.
ImmigrationQA: source-grounded QA dataset of 17,058 pairs across 13 U.S. immigration law subdomains. Fine-tuned Llama 3.2 3B with LoRA on corpus of 10,056 validated documents. Fine-tuned model: 1.08/3.0 (16.8% fully correct) vs Llama 3 8B base: 0.85/3.0 (4% fully correct), 27% relative improvement. Cost: ~$29. Dataset, model, and code publicly released.
Researchers reveal that statistical watermarks in LLMs are vulnerable to linear ensembles. Averaging probability distributions across 3-5 models cancels out watermark perturbations. WASH (Watermark Attenuation via Statistical Hybridisation) defeats detection across 6 watermarking schemes, reducing z-scores from 5-300 to <2 (threshold: 4), while improving output quality by 27.5%.
Item Response Theory-based method detects mislabels in 7 LLM benchmarks at 95% precision on top 200 examples across 114 models. Analysis reveals errors from mechanical labeling heuristics, inherited annotation mistakes, and fundamentally ambiguous items. Reward models specialize in stylistic preference over factual knowledge; one frontier model agrees with detected mislabels at 78% accuracy versus 38% for peers.