Reinforcement learning is a method where an agent learns by receiving rewards or penalties based on its actions. DeepMind's AlphaGo used it to defeat the world's top Go players.
Theoretical study of optimal learning rates in 2-3 layer linear neural networks. Derives exact closed-form expressions for gradients and test loss after 1-2 gradient descent steps. Key finding: unequal layer-wise learning rates minimize loss in early training, while equal rates become optimal later. Code released.
Asymptotic theory for SGLD-Gibbs combining stochastic gradient Langevin dynamics with Gibbs updates for Bayesian inference in latent variable models. Authors derive jump-diffusion limits and propose hyperparameter tuning guidance ensuring statistically meaningful uncertainty quantification.
Theoretical study showing how transformers learn to implement tree search (DFS) via RL. A two-head transformer naturally emerges from policy gradient training on stochastic trees without expert demonstrations. The model generalizes to unseen depths and adapts its strategy based on goal distributions.
New arXiv paper proposing discrete-time approximations to SG(L)D with and without momentum, enabling accurate predictions of stationary covariance and integrated autocorrelation time. Non-asymptotic error bounds for practical tuning and uncertainty quantification, validated on misspecified models and large batch sizes.
ReMax formalizes exploration in RL through retries: a policy is evaluated by expected maximum return over M samples. Exploration emerges naturally without explicit bonuses. RePPO, a PPO variant optimizing ReMax, generalizes discrete M to continuous parameter m for fine-grained exploration control. Results on MinAtar and Craftax benchmarks.
CSRP, a three-stage framework for Chinese grammatical error correction, combines continual pre-training (5.9M samples), Chain-of-Thought fine-tuning, and policy optimization with efficiency-aware rewards. Achieves 50.99 F₀.₅ on NACGEC and outperforms GPT-4 on spelling correction (59.61 F1).
ProactiveLLM learns when to interact with streaming inputs without external signals. Through monotonic random masking and synchronized privileged self-distillation, the model perceives semantic sufficiency from partial inputs. Reduces interaction latency while maintaining quality on text and speech streaming tasks.
TOPD (Trajectory-aware On-Policy Distillation) improves LLM reasoning by using near-future trajectory information to identify truly divergent states. On AIME24/25, TOPD reaches 63.3%/53.3% vs 60.0%/46.7% in standard OPD, showing 30% of high-loss tokens are false positives.
New automated metric (Triangulated Preference Shift score) to measure lexical biases introduced during preference learning (RLHF) in LLMs without manual curation. Analysis across 6 model families reveals a shift toward a 'language of prestige' (overuse of 'delve', 'furthermore').
SelSkill, a dual-granularity preference-learning framework, optimizes selective skill invocation in agentic tasks. On ALFWorld with Qwen3-8B: +10.9 pp task success, +29.1 pp execution precision. On BFCL: +5.7 pp task success, +29.5 pp execution precision. Zero-shot transfer to Tau-bench and PopQA.
SPADER is an RL framework for tool-augmented LLM agents in Multi-Answer QA. It introduces Step-wise Peer Advantage (SPA) for fine-grained credit assignment over long trajectories, and a diversity-aware exploration reward promoting rare entity discovery. Evaluated on QAMPARI, Mintaka, WebQSP, QUEST: improves recall and F1 vs prompting and supervised RL baselines.
Demo2Reward optimizes VLM reward model language instructions at test-time using 3-10 expert demonstrations to reduce false positives in robotics. No additional training required. Validated on simulated tasks and real-world transfer.
CTRL-STEER introduces a closed-loop control framework for Vision-Language-Action (VLA) models. Instead of fixed steering coefficients, it adaptively adjusts intervention strength over time using PID or reinforcement learning controllers. Experiments on OpenVLA with LIBERO task suites demonstrate improved concept regulation stability and better steering-task success trade-offs without retraining the base model.
New set-distance reward method for reinforcement learning of vision-language models on chest X-ray report generation. Tested on Qwen3-VL, Gemma3 with GRPO: 6.80% (BERTScore), 7.82% (RadGraph F1), 4.45% (CheXbert F1) improvements over supervised fine-tuning. Enables test-time best-of-N selection and mid-generation pruning reducing tokens by 50%.
RAFT is a two-stage domain fine-tuning method that mitigates catastrophic forgetting. It refines data via self-conditioned rewriting and answer fusion, then applies on-policy distillation where the original model provides soft targets on student-generated trajectories. Across five domains, RAFT improves domain accuracy by 23.2% over standard SFT and recovers 18.2% of degradation on MS-Bench.
Iterative AI workflow optimizes graphite-based anodes through sequential learning and experimental feedback loops. Citrine Platform generates surrogate models and refines manufacturing constraints. Results: fabrication reliability improved from frequent failures to 100% success, cells ≥350 mAh/g increased from 28.4% to 84.8%, capacity retention rose from 42.1% to 97.3%.
LithoGRPO combines flow matching with GRPO-based reinforcement learning to optimize lithography masks in semiconductor manufacturing. The framework integrates explicit physics-based reward functions and proposes a fast shot-counting algorithm achieving 130x speedup. State-of-the-art results over optimization and learning-based methods.
ARCA introduces a token-level credit assignment method for LLM reinforcement learning that addresses degeneracy of intrinsic signals (surprisal, entropy reduction, policy divergence) under LoRA. It measures adapter salience directly via L2 norm of hidden-state residuals instead of output-distribution shifts. Tested on MATH/Qwen3-1.7B with GRPO, ARCA avoids pathological weight concentration.
CHAM-net, a contrastive hierarchical adaptive meta-network, predicts global methane flux by capturing site-specific dynamics through a conditioned encoder-decoder architecture. On simulation and observational datasets, the model achieves nRMSE=0.43 (emissions) and R²=0.97, outperforming all baselines.
MERIT, a multi-horizon memory retrieval framework for text-to-SQL agents, maintains episode-level and turn-level memory with learned retrieval policies optimized via reinforcement learning. On BIRD-Interact, it outperforms no-memory and static-retrieval baselines, improving success rate while reducing interaction turns.
Delayed per-step reward attribution method for training LLM agents in multi-agent strategic interaction. An 8-billion-parameter open-source model trained with this approach matched or surpassed GPT-5 and won both Open and Efficient tracks at MindGames Arena benchmark (NeurIPS 2025).
CAST is an answer-free self-distillation method for GRPO (Group Relative Policy Optimization). It uses a stop-gradient self-teacher to shape token-level advantages by trajectory correctness, with bidirectional advantage sign reversal and bounded advantages for zero-variance groups. Improves mathematical reasoning performance.
MindZero is a self-supervised reinforcement learning framework training multimodal LLMs to infer human mental states without annotations. The model is rewarded for generating mental state hypotheses that maximize the likelihood of observed actions. After training, inference becomes fast single-pass and outperforms model-based methods in both accuracy and efficiency.
Modern LLMs systematically overestimate their competence and attempt unsolvable queries. Researchers propose Capability Self-Assessment (CSA), formulated as a policy-learning problem using reinforcement learning, to teach models to recognize their limits. RL significantly outperforms supervised fine-tuning, preserves original capabilities, and generalizes out-of-distribution.
Novel shielding framework for RL agents ensuring formal safety guarantees in MDPs with unknown transition dynamics. Uses robust MDPs (RMDPs) with sets of transition probabilities and LTL formulas. Combines shielding with PAC-learning methods to construct minimally restrictive shields while guaranteeing safety.
Parameterized Diffusion Policy (PDP) conditions diffusion policies on continuous parameters embedded in a learned behavior manifold. The approach enables smooth interpolation between strategies and efficient adaptation to novel constraints without weight updates. Improved performance on multimodal benchmarks in simulated and real-robot experiments.
Study of effectiveness and efficiency of tool-calling in LLM agents. Authors show evaluation pipelines are sensitive to minor choices (random seed, system prompt, multi-turn templates) affecting leaderboard reliability. They identify two sources of computational waste in RL and propose two acceleration techniques without performance degradation.
Preference Delta Aggregation (PDA) aggregates weak preference signals from model pairs (e.g., Qwen3 4B vs 1.7B) via LoRA merging. Geometric Alignment Merging (GAM) aligns adapter subspaces before aggregation. On knowledge reasoning and agentic search benchmarks, PDA+GAM improves Qwen3 8B by +6.8 and +7.3 points respectively.
OPCD method to improve large models using weak critics. Instead of weak supervisors as labelers, they guide revisions. Progressive on-policy critique distillation filters high-quality critiques and distills critic-guided behavior into strong models via adaptive self-teacher signals. Results on reasoning and alignment benchmarks.
A researcher who spent months inside verl (ByteDance's RL post-training framework) documents its internals: RLHF loop orchestration, single-controller pattern, data structures (DataProto), and a NCCL bug discovered. Abandoned fork but knowledge shared with the community.
Turing Award winner Richard Sutton argues pure generative AI lacks a critical ability: self-evaluation of results. Without it, genuine scientific discovery remains impossible. Only built-in evaluation loops, as seen in AlphaGo and AlphaProof, enable true AI creativity.
Discussion on fine-tuning small LLMs with annotated conversational data including reasoning traces and tool-calling decisions. Author proposes structuring data as samples with full history and loss masking on non-assistant tokens. Asks whether SFT is sufficient or if RL (PPO, GRPO, DPO) is needed to optimize tool use.
CSRM (Configurable Safety Reward Model) jointly optimizes calibrated safety compliance and reward modeling to adapt LLMs to heterogeneous and evolving safety requirements. Achieves 94.6% F1 on CoSApien and 75.8% F1 on DynaBench without additional human annotation.
GRiD, a diffusion-model framework, generates graph-like rules for knowledge graph reasoning. Combines supervised pre-training and reinforcement learning to discover complex rules (cycles, branches) beyond simple chains. Evaluated on 6 benchmarks with open-source code.
Method to automatically generate fine-grained evaluation rubrics without human annotation, tested on four benchmarks. Training-free approach, then iterative fine-tuning via meta-judge reward signals. A fine-tuned 14B rubric generator outperforms larger proprietary models.
SLAT is an RL framework that reduces redundancy in chain-of-thought reasoning by selectively suppressing low-utility segments. On standard benchmarks, the method achieves 50% reasoning length reduction while maintaining competitive accuracy.
New post-training method for reasoning models: Feedback Distillation trains the model to match its own distribution conditioned on LLM-generated feedback at token level. Tested on Lean4 theorem-proving, it maintains greater trajectory diversity than GRPO, improves policy entropy and pass@k scaling. Combined with GRPO, it outperforms either method alone.
Novel transformer-based architecture for autonomous resource management in heterogeneous satellite clusters (optical and SAR). Uses model-free reinforcement learning for real-time decision-making in Earth Observation missions. Demonstrates significant performance improvements and transferability across varying cluster sizes.
ElasticMem introduces a learnable latent memory framework for LLM agents with adaptive retrieval and elastic budget allocation via learned policy. On Qwen2.5-3B and 7B backbones, achieves 26.2% and 24.6% QA accuracy gains, 66.3% and 27.2% ALFWorld success improvements, with lowest token cost.
arXiv study on iterative refinement of LLM-generated reward functions for sparse structured RL. Authors identify two dominant failure modes (reward flooding, semantic misunderstanding) and propose diagnostic-driven refinement guided by failure-mode taxonomy. Results: DoorKey-8x8 improves from 2.3% to 97.6%, KeyCorridor from 31.2% to 86.7%. Limitations: method restricted to PPO and sparse structured tasks.