Theoretical study of optimal learning rates in 2-3 layer linear neural networks. Derives exact closed-form expressions for gradients and test loss after 1-2 gradient descent steps. Key finding: unequal layer-wise learning rates minimize loss in early training, while equal rates become optimal later. Code released.
Asymptotic theory for SGLD-Gibbs combining stochastic gradient Langevin dynamics with Gibbs updates for Bayesian inference in latent variable models. Authors derive jump-diffusion limits and propose hyperparameter tuning guidance ensuring statistically meaningful uncertainty quantification.
InfoAtlas is a foundation model that estimates mutual information between high-dimensional variables in a single forward pass, eliminating costly iterative optimization. Pretrained on synthetic data with diverse dependence patterns, it achieves 100× speedup over neural MI estimators while matching accuracy and generalizes to complex real-world scenarios.
Theoretical study showing how transformers learn to implement tree search (DFS) via RL. A two-head transformer naturally emerges from policy gradient training on stochastic trees without expert demonstrations. The model generalizes to unseen depths and adapts its strategy based on goal distributions.
Novel spatio-temporal graph anomaly detection method modeling spectral energy shifts. Detects camouflaged anomalies (decreased spectral variation) missed by existing spectral approaches. Energy-aware framework compatible with message passing, no specialized sequence modules required.
New arXiv paper proposing discrete-time approximations to SG(L)D with and without momentum, enabling accurate predictions of stationary covariance and integrated autocorrelation time. Non-asymptotic error bounds for practical tuning and uncertainty quantification, validated on misspecified models and large batch sizes.
ChurnNet compares classical methods (Random Forests, XGBoost, SVM) with a Unified Multi-Task Time Series Model for customer churn prediction. Results show traditional approaches outperform the modern AI model in predictive performance, data efficiency, and computational requirements across multiple datasets.
ReMax formalizes exploration in RL through retries: a policy is evaluated by expected maximum return over M samples. Exploration emerges naturally without explicit bonuses. RePPO, a PPO variant optimizing ReMax, generalizes discrete M to continuous parameter m for fine-grained exploration control. Results on MinAtar and Craftax benchmarks.
Study on adaptive data selection for wearable prediction across 3 modalities (heart rate, activity, EMA) in longitudinal dataset. AUROC gains up to 0.7 for low-baseline participants, strong inverse correlation (r=-0.67) between improvement and baseline performance. Limited benefit for strong baselines.
Post-training quantization (PTQ) reduces reasoning model accuracy and increases chain-of-thought length. 52% of failures involve correct intermediate answers not output as final answers. A training-free logit penalty on overthinking markers ("wait", "but", "alternatively") reduces CoT length by 12-23% while preserving accuracy across 5 models (1.5B-32B) and 5 benchmarks.
BitsMoE introduces spectral-energy-guided bit allocation for MoE LLM quantization. Using SVD decomposition, it preserves shared basis unquantized and fine-grained quantizes expert-specific factors via integer linear programming. On Qwen3-30B at 2-bit, it improves accuracy by 27.83 percentage points and increases decoding speed 1.76× over GPTQ.
Hoeffding Concept Bottleneck Models (HCBM) replace linear concept aggregation with non-linear sparse decomposition based on gradient-boosted trees. Robust to inter-concept leakage, they improve explainability of computer vision predictions, particularly on overhead imagery.
Demo2Reward optimizes VLM reward model language instructions at test-time using 3-10 expert demonstrations to reduce false positives in robotics. No additional training required. Validated on simulated tasks and real-world transfer.
Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) compress neural network weights using compact core tensors trained via reverse-mode automatic differentiation. Tested on AlexNet and VGG-16, they achieve per-layer compression ratios from 2000× to 77000× while maintaining or improving accuracy.
GEM is a concept erasure framework for Rectified Flow Transformers. It bridges trajectory-based unlearning (Generative Flow Networks) and teacher-guided erasure, using geometric guidance signals to suppress unwanted concepts while preserving benign generation and preventing harmful content synthesis.
BudgetDraft trains a sparse drafter for speculative decoding in long-context inference (4K-16K tokens). The method exposes the model to multiple KV budgets during training and aligns each sparse view with a shared full-cache teacher target. Results: 6.55x, 4.46x, 2.10x speedup vs autoregressive decoding at 4K, 8K, 16K tokens.
RAFT is a two-stage domain fine-tuning method that mitigates catastrophic forgetting. It refines data via self-conditioned rewriting and answer fusion, then applies on-policy distillation where the original model provides soft targets on student-generated trajectories. Across five domains, RAFT improves domain accuracy by 23.2% over standard SFT and recovers 18.2% of degradation on MS-Bench.
Novel approach for Major Depressive Disorder detection from EEG without data augmentation. SGC (Score-Guided Classification) uses an unsupervised generative network to model pathological anomalies as prior, fused with deep feature representations. Cross-Channel Spatial Adaptation module handles multi-center channel heterogeneity. Validated on Mumtaz2016 and MODMA datasets.
Iterative AI workflow optimizes graphite-based anodes through sequential learning and experimental feedback loops. Citrine Platform generates surrogate models and refines manufacturing constraints. Results: fabrication reliability improved from frequent failures to 100% success, cells ≥350 mAh/g increased from 28.4% to 84.8%, capacity retention rose from 42.1% to 97.3%.
arXiv paper proposing principled approaches for designing practical LLM-based agentic systems. Framework with "pseudo-tools" recursively calling LLMs on restricted context. Hand-engineered fixed workflows are cheaper and more accurate than dynamically-planned workflows. Learning methods for pseudo-tools and fixed workflows outperform hand-engineered agents.
PRAXIS is an algorithm that efficiently computes Rashomon sets (sets of near-optimal models) for sparse decision trees. The approach drastically reduces memory and runtime while recovering most of the full set. Code available on GitHub.
LithoGRPO combines flow matching with GRPO-based reinforcement learning to optimize lithography masks in semiconductor manufacturing. The framework integrates explicit physics-based reward functions and proposes a fast shot-counting algorithm achieving 130x speedup. State-of-the-art results over optimization and learning-based methods.
Study of delayed generalization (grokking) during language model pre-training. Using an exposure-based framework and BLiMP minimal pairs, authors observe delayed generalization across five grammatical phenomena. Grammatical concept vectors become more predictive after generalization and occupy higher-dimensional subspaces.
ARCA introduces a token-level credit assignment method for LLM reinforcement learning that addresses degeneracy of intrinsic signals (surprisal, entropy reduction, policy divergence) under LoRA. It measures adapter salience directly via L2 norm of hidden-state residuals instead of output-distribution shifts. Tested on MATH/Qwen3-1.7B with GRPO, ARCA avoids pathological weight concentration.
InfoNCE, the standard contrastive learning objective, relies on a statistical assumption misaligned with modern normalized embeddings. Authors propose WEINCE, a simple modification using extreme value theory and anchor-wise online batch statistics to correct softmax logits without additional parameters. Consistent improvements across 5 vision benchmarks.
New quantization method preserving inner products with unseen vectors. Adaptive unbiased algorithms developed with theoretical guarantees. Practical implementations 2-10× faster than prior ASQ state-of-the-art while maintaining quality.
Masked diffusion models: a lightweight policy network learns optimal token unmasking order. Loss reweighting by denoiser probabilities. Outperforms heuristics on order-sensitive tasks like combinatorics and proteins.
FLaG is a lightweight hallucination detection framework for LLMs that models correctness through latent evidence groups. Using energy-based routing and log-marginal aggregation, it captures heterogeneous hallucination patterns without modifying the underlying model. SOTA results across multiple benchmarks with robust transfer across datasets.
arXiv paper on temperature's role in LLM distillation. Authors show forward KL (FKL) outperforms reverse KL (RKL) at higher temperatures, contrary to prior empirical conclusions that omitted this parameter. Temperature enriches FKL with non-dominant token signals while merely rescaling RKL gradients.
Online, distribution-free framework for controlling Conditional Value-at-Risk (CVaR) in non-stationary and adversarial environments. Combines conformal tail risk control, online learning, and Rockafellar-Uryasev variational representation. Provable safety guarantees for nonlinear tail risk under arbitrary data-generating processes. Applications: portfolio risk management and LLM toxicity mitigation.
Perturbative approach for nonparametric instrumental variable (NPIV) estimation inspired by perturbation theory in physics. Extends kernel ridge methods with systematic higher-order corrections. Reduces prediction error by up to 99% in high-dimensional ill-defined cases (β > 0.7) compared to standard ridge regression.
KG-Guard detects hallucinations in knowledge base question answering (KBQA) systems using an augmented graph and lightweight encoder. The model achieves F1 scores of 82.0–87.4 on WebQSP/ComplexWebQuestions with 305× fewer parameters than baselines, and improves downstream KBQA F1 by 13–14.5 points through iterative refinement feedback.
CHAM-net, a contrastive hierarchical adaptive meta-network, predicts global methane flux by capturing site-specific dynamics through a conditioned encoder-decoder architecture. On simulation and observational datasets, the model achieves nRMSE=0.43 (emissions) and R²=0.97, outperforming all baselines.
Study of effectiveness and efficiency of tool-calling in LLM agents. Authors show evaluation pipelines are sensitive to minor choices (random seed, system prompt, multi-turn templates) affecting leaderboard reliability. They identify two sources of computational waste in RL and propose two acceleration techniques without performance degradation.
FoLoRA, a forgetting-aware optimization framework, adapts foundation models via generalized Rayleigh quotient. It balances target task performance and preservation of pretraining capabilities by penalizing adaptation directions with low utility relative to forgetting. Tested on math, code, and instruction-following adaptation.
Unified Gait2Hip-60 benchmark comparing LSTM, Transformer, and Mamba to predict hip muscle forces and joint moments from gait kinematics. Transformer outperforms other models (R²=0.819 for forces, R²=0.862 for moments). External validation on 9 femoral head osteonecrosis patients shows moderate generalization (R²=0.537–0.569).
Multi-model study (Pythia-1.4B, Gemma-2, Qwen2.5-7B, Llama-3.1-8B) on linear representations of synthetic dishonesty. Linear probes detect deception with AUC ≥0.99 as early as layers 1-3. Dishonesty representations consolidate progressively in deeper layers, with implications for activation-based monitoring.
NumLeak measures memorization of public benchmarks in frontier LLMs. Models recall Fama-French data (r=0.97-0.99), US unemployment, and NOAA temperature with high fidelity. On recent unseen data, parse rate drops to 21-57% but r stays ~0.99 for answered months. A one-line system-prompt defense blocks 99.8% of attacks.
LongDS-Bench evaluates AI agents' ability to maintain analytical context over long horizons. The benchmark contains 68 multi-turn data analysis tasks (2,225 turns) from real Kaggle notebooks. Best models reach only 48.45% accuracy, with a 47-point performance drop from early to late turns. Long-horizon errors account for 52–69% of failures.
Formal study of calibration for probabilistic label ranking. Authors define a hierarchy of notions (full rankings, sub-rankings, top-k) and show popular models are poorly calibrated. Application to RLHF reward models reveals calibration and accuracy are not perfectly correlated.
A new counterfactual evaluation metric (CSS) reveals that six frontier models ranked similarly on traditional coverage-based metrics rank in nearly opposite order when assessed on their ability to update clinical recommendations in response to oncology case mutations. All models fail on surgery-status interventions, a safety blind spot invisible to coverage metrics.
VeriGate extends GRPO by combining verifier rewards with step-level supervision. The method uses a Process Reward Model (PRM) to assign fine-grained credit to tokens, avoiding gradient collapse when all trajectories receive identical rewards. On MATH with Qwen2.5-Instruct (1.5B/7B), VeriGate improves accuracy by ~20% and ~12% respectively.
DisjunctiveNet introduces a neuro-symbolic framework to embed hard mixed-integer linear constraints and logical rules directly into neural networks using differentiable optimization layers. Through hierarchical convex relaxations, the approach ensures exact rule satisfaction while maintaining strong predictive performance on real-world datasets.
New method to identify whether a dataset was used in model training by analyzing semantic correlation descriptors (SCDs) learned internally. White-box approach outperforms black-box baselines (RMIA, LiRA) with gains up to 60% ROC-AUC on NLI, emotion, and medical text classification tasks.
First model extraction attack targeting graph neural networks exploiting explainability interfaces. Method uses binary explanation masks and Monte Carlo edge sensitivity estimation to bypass strict black-box constraints. Code released on GitHub.
Theoretical paper on universal multiclass transductive online learning with unbounded label space. Characterizes learnability: only two possible optimal rates (bounded or logarithmic). Introduces LCLL tree combinatorial structure and extends results to agnostic and stochastic settings.
GC-MoE is a graph-conditioned mixture-of-experts framework for spatio-temporal traffic forecasting. It assigns each node a personalized combination of frozen GNN experts based on graph topology and recent input windows. Trained on PEMS04, PEMS07, METR-LA, and PEMS-BAY, the model improves MAE with only ~17K trainable parameters on top of 1.5M frozen expert weights.
Theoretical study of padded transformer expressivity. Authors prove polynomially padded constant-precision transformers are equivalent to L-uniform AC⁰, while growing-precision ones achieve L-uniform TC⁰. Model depth and numeric precision are key factors; width beyond logarithmic does not increase expressivity.
Study of stylistic signatures introduced by LLM alignment. Researchers show post-training creates a detectable AI-like style. They propose PASTA, a training-free method that localizes and ablates this signature during decoding, reducing detection rates across 11 aligned models and 6 AI detectors.
Study on long-term effects of data selection during multi-stage LLM fine-tuning. Authors show that short-term optimal strategies (loss-based, gradient-based, diversity-based) can slow future learning and increase catastrophic forgetting. They propose LHAS (Long-Horizon Aware Selection) to evaluate selection as a global training intervention.