Part II. Training, Learning and Data

Delta Learning Hypothesis

Internal Concept ID AIKC-000068 · 1 record in 2025

Delta Learning Hypothesis — AIK-2025-S-P-DELTALEARN

Type / Origin Parent Date of Appearance Original publication
S / P L5-D8 2025-07-08 2025-12-05

L5-D8 — Delta Learning Hypothesis (type S)

Date of Appearance (documented milestone, not an exhaustive first use): July 8, 2025 (Geng et al., arXiv:2507.06187v1)
Public Entry ID: AIK-2025-S-P-DELTALEARN · Internal Concept ID: AIKC-000068
Concept: C:DELTALEARN

As published: Table, A-016. Core Concept: A data curation approach that focuses on learning from the differences between weak and weaker data, rather than from absolute quality. Why It's a 2025 Trend: Discussed at ICML 2025 as an alternative to the "more data is always better" axiom, especially for preference learning and alignment. Source Context: ICML 2025 Attendee Reflection

Source (corrected record): Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., & Koh, P. W. (2025). The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains. COLM 2025. arXiv:2507.06187. [C03]

Editorial note: The paper was published at COLM 2025. The collection did not verify the table's "discussed at ICML 2025".

Distillation at Scale / Student-Teacher Cascades

Internal Concept ID AIKC-000056 · 1 record in 2025

Distillation at Scale / Student-Teacher Cascades — AIK-2025-S-P-DISTILSCALE

Type / Origin Parent Date of Appearance Original publication
S / P L1-S10 — (not established) 2025-11-30

L1-S10 — Distillation at Scale / Student-Teacher Cascades (type S)

Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-DISTILSCALE · Internal Concept ID: AIKC-000056
As published: Distillation at Scale / Student-Teacher Cascades Definition: Large-scale knowledge distillation methods that compress capability into smaller models or chains of models while retaining alignment properties. Why 2025: Practical route to democratize capabilities: distilling multi-modal, tool-using policies into deployable students became common. Example: Distilling a tool-enabled agent into a smaller model for mobile use.

Drop-Upcycling (Sparse MoE Training)

Internal Concept ID AIKC-000040 · 1 record in 2025

Drop-Upcycling (Sparse MoE Training) — AIK-2025-S-P-DROPUPCYCLE

Type / Origin Parent Date of Appearance Original publication
S / P L1-M2 2025-02 2025-11-30

L1-M2 — Drop-Upcycling (Sparse MoE Training) (type S)

Date of Appearance: February 2025 (Nakamura et al., arXiv:2502.19261; ICLR 2025)
Public Entry ID: AIK-2025-S-P-DROPUPCYCLE · Internal Concept ID: AIKC-000040
Concept: C:DROPUPCYCLE

As published: Definition: A training strategy where experts in a sparse MoE are partially reinitialized during upcycling (dense → sparse), improving specialization and avoiding catastrophic forgetting. Why 2025: Became the first robust procedure to consistently stabilize MoE upcycling. References: • Nakamura, T., Akiba, T., Fujii, K., et al. Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization. arXiv (2025).

Why 2025 (corrected record): Proposed by Nakamura et al. (arXiv:2502.19261; ICLR 2025), who report that it outperforms previous MoE construction methods over long training. [C05]

Dynamical Regularization (in Diffusion)

Internal Concept ID AIKC-000065 · 1 record in 2025

Dynamical Regularization (in Diffusion) — AIK-2025-S-P-DYNREG

Type / Origin Parent Date of Appearance Original publication
S / P L5-D5 — (not established) 2025-12-05

L5-D5 — Dynamical Regularization (in Diffusion) (type S)

Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-DYNREG · Internal Concept ID: AIKC-000065
Concept: C:DYNREG

As published: Table, A-013. Core Concept: The theoretical finding that diffusion models have an inherent, time-delayed memorization phase, creating a window for effective generalization. Why It's a 2025 Trend: A NeurIPS 2025 Best Paper unified theory and practice, providing a foundational explanation for why diffusion models generalize well. Source Context: NeurIPS 2025 Best Paper

E-Processes / Game-Theoretic Statistics

Internal Concept ID AIKC-000070 · 1 record in 2025

E-Processes / Game-Theoretic Statistics — AIK-2025-E-P-EPROCESS

Type / Origin Parent Date of Appearance Original publication
E / P L5-D10 — (not established) 2025-12-05

L5-D10 — E-Processes / Game-Theoretic Statistics (type E)

Date of Appearance: Not established. Game-theoretic statistics predates 2025 (Shafer and Vovk, as the full entry notes)
Public Entry ID: AIK-2025-E-P-EPROCESS · Internal Concept ID: AIKC-000070
Concept: C:EPROCESS; the headword is renamed "Game-Theoretic Statistics" in the full entry

As published: Table, A-018. Core Concept: A framework for statistical inference based on betting metaphors and non-negative martingales, allowing for valid inference at any data-checking point. Why It's a 2025 Trend: The subject of a well-received ICML 2025 tutorial, offering an alternative to p-values for sequential/adaptive analysis. Source Context: ICML 2025 Tutorial

Source (corrected record): Ramdas, A. (2025, July 14). Game-theoretic Statistics and Sequential Anytime-Valid Inference. Tutorial, ICML 2025, Vancouver. https://icml.cc/virtual/2025/tutorial/40002 [C02]

Editorial note: The full entry is below (C02).

Corrected full entry — Game-Theoretic Statistics (L5-D10)

Revised entry by DeepSeek-V3.2 (corrected canonical text; C02 applied). Source text: A-110 to A-116.

Definition

A framework for sequential and anytime-valid inference that reimagines statistical testing as a gambling game. It uses e-values and e-processes (non-negative martingales that represent a gambler's wealth) to provide statistical evidence that remains valid regardless of when or why you stop collecting data (a practice known as "optional stopping").

2025 Context

While building on foundational work by Shafer, Vovk, and others, this framework saw renewed emphasis and discussion in 2025 for its application to modern AI challenges. It was the subject of an ICML 2025 tutorial, Aaditya Ramdas's "Game-theoretic Statistics and Sequential Anytime-Valid Inference" (July 14, 2025), which presented it as an alternative to classical p-values, which can be invalidated by the common practice of adaptively analyzing data. [C02]

Implication

Game-Theoretic Statistics addresses a critical need in machine learning and science, where data often arrives sequentially and analyses are performed adaptively. By providing robust statistical guarantees for continuous monitoring and adaptive experimentation, it offers a more reliable foundation for A/B testing, safety monitoring of deployed AI systems, and reinforcement learning, where decisions must be made under uncertainty with streaming data.

(The original reads "It was highlighted in sessions at major conferences like ICML 2025 as a compelling alternative to classical p-values". That wording replaced a citation of the ICML 2025 tutorial after a review called it unverifiable. The tutorial existed, so the citation is restored: Ramdas, ICML 2025, July 14, 2025, https://icml.cc/virtual/2025/tutorial/40002.) [C02]


Federated & On-Device Model Training

Internal Concept ID AIKC-000057 · 1 record in 2025

Federated & On-Device Model Training — AIK-2025-S-P-FEDONDEVICE

Type / Origin Parent Date of Appearance Original publication
S / P L1-S12 — (not established) 2025-11-30

L1-S12 — Federated & On-Device Model Training (type S)

Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-FEDONDEVICE · Internal Concept ID: AIKC-000057
As published: Federated & On-Device Model Training Definition: Training and adaptation methods that keep data local (federated learning, split learning) or run inference/adaptation on device for privacy and latency. Why 2025: Regulatory and privacy pressures plus edge compute improvements increased adoption for personalization and compliance. Example: On-device personalization of a language assistant with differential privacy guarantees.

Generative Feedback Loop (Self-Adaptation)

Internal Concept ID AIKC-000033 · 1 record in 2025

Generative Feedback Loop (Self-Adaptation) — AIK-2025-S-P-GENFEEDBACK

Type / Origin Parent Date of Appearance Original publication
S / P L2-E8 — (not established) 2025-12-01

L2-E8 — Generative Feedback Loop (Self-Adaptation) (type S)

Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-GENFEEDBACK · Internal Concept ID: AIKC-000033
Concept: C:GENFEEDBACK

As published: • Concept: An agent's capability to use its own generated data (e.g., task failure traces, environment observations) to self-diagnose and automatically update its policy or world model without human intervention or re-training from scratch. • Why 2025: Breakthroughs in agent reliability for real-world tasks, enabling full self-correcting loops for complex software systems. Reference (table, A-063): Li, M., et al. Enabling Self-Correction: Generative Feedback Loops for Policy Adaptation in Autonomous Agents. Type & Source: Conference (ICRA 2025)

Reference (corrected record): None. The cited "Enabling Self-Correction" paper (Li, M., et al., ICRA 2025) could not be found, and the item is unsourced. [C04]

High-Dimensional Learning Dynamics

Internal Concept ID AIKC-000072 · 1 record in 2025

High-Dimensional Learning Dynamics — AIK-2025-S-P-HDLD

Type / Origin Parent Date of Appearance Original publication
S / P L5-D12 — (not established) 2025-12-05

L5-D12 — High-Dimensional Learning Dynamics (type S)

Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-HDLD · Internal Concept ID: AIKC-000072
Concept: C:HDLD

As published: Table, A-020. Core Concept: The study of learning algorithms in the regime where both sample size and model parameters are large, using tools from statistical physics. Why It's a 2025 Trend: Focus of the 3rd annual HiLD workshop at ICML 2025, crucial for explaining phenomena in large neural networks. Source Context: ICML 2025 Workshop (HiLD)


Memorization Sinks

Internal Concept ID AIKC-000063 · 1 record in 2025

Memorization Sinks — AIK-2025-S-P-MEMSINK

Type / Origin Parent Date of Appearance Original publication
S / P L5-D3 2025-07-14 2025-12-05

L5-D3 — Memorization Sinks (type S)

Date of Appearance (documented milestone, not an exhaustive first use): July 14, 2025 (Ghosal, Maini & Raghunathan, arXiv:2507.09937v1)
Public Entry ID: AIK-2025-S-P-MEMSINK · Internal Concept ID: AIKC-000063
Concept: C:MEMSINK

As published: Table, A-011. Core Concept: A training technique that isolates a model's memorization of specific data examples into dedicated, sparsely activated neurons. Why It's a 2025 Trend: Proposed as a method to improve generalization by separating memorized data from general concepts within a model's architecture. Source Context: ICML 2025 Attendee Reflection

Source (corrected record): Ghosal, G. R., Maini, P., & Raghunathan, A. (2025). Memorization Sinks: Isolating Memorization during LLM Training. ICML 2025. arXiv:2507.09937. [C03]

Nested Learning (Continual AI)

Internal Concept ID AIKC-000028 · 1 record in 2025

Nested Learning (Continual AI) — AIK-2025-S-P-NESTEDLEARN

Type / Origin Parent Date of Appearance Original publication
S / P L2-E3 — (not established) 2025-12-01

L2-E3 — Nested Learning (Continual AI) (type S)

Date of Appearance: Not established. The installment dates the paper to NeurIPS 2025
Public Entry ID: AIK-2025-S-P-NESTEDLEARN · Internal Concept ID: AIKC-000028
Concept: C:NESTEDLEARN. Cross-year link to a 2026 entry, "Aikipedia: Nested Learning" (post 14472, January 3, 2026). This 2025 record is unchanged; the 2026 collection links its record to this concept

As published: • Concept: A novel machine learning paradigm where knowledge is stored in a hierarchical, "nested" structure, allowing the model to continually learn new tasks without suffering from catastrophic forgetting of old knowledge. • Why 2025: Formalized with a dedicated paper at NeurIPS 2025, bridging the long-standing gap in practical, scalable continual learning. Reference (table, A-058): Behrouz, A., et al. Nested Learning: The Illusion of Deep Learning Architectures. Type & Source: Conference (NeurIPS 2025)

Editorial note: The concept also has a 2026 entry. The 2025 original stays. Neither audit flagged the reference.

On-Policy Distillation

Internal Concept ID AIKC-000025 · 1 record in 2025

On-Policy Distillation — AIK-2025-F-P-ONPOLDISTIL

Type / Origin Parent Date of Appearance Original publication
F / P standalone 2024-01-17 2025-10-29

On-Policy Distillation

Corrected collection text. The original publication is preserved unchanged in the Historical Lexicon Archive.

Field Value
Term On-Policy Distillation
Date of Appearance January 17, 2024 (Agarwal et al., arXiv:2306.13649v3, the first version titled "On-Policy Distillation of Language Models"). Previously named GKD ("Generalized Knowledge Distillation"): the method first appeared under that name on June 23, 2023 (arXiv:2306.13649v1). Earliest documented use identified; not an exhaustive priority claim
Date of Original Publication October 29, 2025 (Champaign Magazine, post 13892)
Historical Coverage Through October 29, 2025, the date of original publication. Corrections rely only on sources available by that date
Original Contributor ChatGPT (GPT-5, as addressed in the post; release not stated)
Original Reviewers Grok (Expert mode; release not stated); Claude Sonnet 4.5
Corrections Prepared by Claude Opus 5.5 (Anthropic)
Independent review DeepSeek-V4.1
Base version Sole published version (post 13892)
Entry ID AIK-2025-F-P-ONPOLDISTIL (concept AIKC-000025)

Markers: [Cnn] marks a corrected or added passage, documented in the entry's correction register; [ST01] marks the rearrangement into AES sections. Unmarked text is the original's own wording (archive lines A-009 to A-091), rearranged into the AES sections [ST01].

1. Definition

On-policy distillation is a training paradigm for large language models (LLMs) where a student model generates trajectories under its own policy (i.e., “on-policy”), and a teacher model provides dense, token-level supervision over those trajectories. The student then updates its parameters to align its probability distribution with the teacher’s, typically via a reverse KL divergence loss. This combines the distributional relevance of on-policy reinforcement learning (RL) with the dense feedback of distillation. (Thinking Machines Lab, 2025) [R1]

2. Origin and Historical Development

On-policy distillation of language models was proposed before 2025. Agarwal et al. introduced Generalized Knowledge Distillation (GKD), which trains a student on its own sampled output sequences using the teacher's token-level probabilities (arXiv, June 23, 2023; ICLR 2024; titled "On-Policy Distillation of Language Models" from January 2024) [P1]. Gu et al.'s MiniLLM distilled large language models by minimizing the reverse KL divergence (arXiv, June 14, 2023; ICLR 2024) [P2]. The Thinking Machines Lab post of October 27, 2025 presents its work as extending these and the Qwen3 team's use of the method [R1] [C02].

3. Conceptual or Technical Framework
3.1 Background

LLM development usually proceeds in three phases:

  1. Pre-training — broad language and knowledge acquisition.
  2. Mid-training — domain adaptation (e.g., code, legal, or medical corpora).
  3. Post-training — behavioral alignment, reasoning refinement, instruction following.

Traditional post-training uses either:

Both have trade-offs:

Approach Strength Weakness
Off-policy SFT/Distillation Dense supervision Distributional mismatch (student sees unseen states)
On-policy RL Realistic trajectories Sparse reward → inefficient learning

On-policy distillation resolves this tension: it trains on the student’s visited states while using dense, per-token feedback from a strong teacher.

3.2 Methodology
  1. Trajectory Generation: The student πθ\pi_\theta samples sequences x1:Tx_{1:T} from its current policy. [C03]

  2. Teacher Evaluation: The teacher πT\pi_T computes log-probabilities logπT(xt∣x<t)\log \pi_T(x_t \mid x_{<t}) for each token generated by the student. [C03]

  3. Policy Alignment: The student minimizes a divergence between its distribution and the teacher’s over the same visited states:

    ℒon-policy=𝔼x1:T∼πθ∑t=1TDKL(πθ(⋅∣x<t)∥πT(⋅∣x<t))\mathcal{L}_{\text{on-policy}} = \mathbb{E}_{x_{1:T} \sim \pi_\theta} \sum_{t=1}^{T} D_{\text{KL}}\!\big(\pi_\theta(\cdot \mid x_{<t}) \,\|\, \pi_T(\cdot \mid x_{<t})\big)

    where DKL(p∥q)=∑xp(x)logp(x)q(x)D_{\text{KL}}(p \,\|\, q) = \sum_x p(x)\log \tfrac{p(x)}{q(x)}. [C03] In practice, the reverse KL encourages the student to sharpen its distribution toward the teacher’s mode while maintaining its own trajectory support.

  4. Gradient Update: Gradients are propagated through the student; the teacher is frozen. Training proceeds over multiple on-policy rollouts. [C07]

3.3 Conceptual intuition
3.4 Figure 1 (original)

The image (post attachment 13898) is kept in the archive. Comparative overview of off-policy, on-policy, and on-policy distillation paradigms. The illustration contrasts how trajectories are generated and supervised across three learning setups for large language models.

4. Applications and Implementations
5. Evidence and Evaluation

From Thinking Machines Lab (2025):

6. Significance and Implications

On-policy distillation elegantly integrates reinforcement learning’s state realism with distillation’s dense supervision. By aligning a student’s on-policy trajectories to a teacher’s per-token distribution, it enables efficient, compute-light behavioral alignment — accelerating the path toward scalable and adaptive language models.

7. Limitations, Debates, and Open Questions
Related Method Similarity Key Difference
Supervised Distillation (off-policy) Uses teacher targets Student samples from teacher trajectories
Reinforcement Learning (on-policy) Uses own trajectories Sparse reward vs dense KL loss
Imitation Learning / DAGGER On-policy expert querying LLM variant uses token-level distributions
See Also

The original entry has none. Related records in this collection (labels only; editorial cross-references): Chain-of-Thought Prompting (AIK-2025-F-P-COTPROMPT), on prompting as distinct from training; RLVR (AIK-2025-F-N-RLVR), on RL post-training. [C06]

References and Sources
Editorial Notes
  1. Status. Independently reviewed and accepted by the editor. The dated record is in Appendix G.
  2. Source-date rule. Every source used to correct the text was available by October 29, 2025. The Thinking Machines post is dated October 27, 2025. Its live page has been revised since: a later note records the retirement of the models it used. The teacher sentence behind C04 was in the October 2025 version.
  3. Production fix (C03). The original's LaTeX did not render. Underscores were read as emphasis, and the KL separator printed as ";|;". The formulas here are typeset with the same content. No mathematical change is made.
  4. Embedded reviews. Grok's review (archive A-106 to A-135) found "no factual errors or fabrications" in a text containing the RLHF comparison corrected by C01. It also named the antecedents omitted by the entry (C02). Claude Sonnet 4.5's review (A-136 to A-199) glosses "reverse KL" as minimizing D_KL(π_T || π_θ), which is the forward KL. The entry's own formula is right (confirmed by Muse Spark 1.3). Both reviews are provenance annotations; neither is part of the entry. Review Applied?: Grok, no changes requested; Claude, not applied (no revised entry follows).
  5. Conflict. The preparer is an Anthropic model, and the post includes a published Claude review. No correction touches that review. The point is recorded as an annotation only.
Credits

Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.

Parameter-Efficient Fine-Tuning (PEFT) evolution

Internal Concept ID AIKC-000053 · 1 record in 2025

Parameter-Efficient Fine-Tuning (PEFT) evolution — AIK-2025-S-P-PEFT

Type / Origin Parent Date of Appearance Original publication
S / P L1-S7 — (not established) 2025-11-30

L1-S7 — Parameter-Efficient Fine-Tuning (PEFT) evolution (type S)

Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-PEFT · Internal Concept ID: AIKC-000053
Concept: C:PEFT ("evolution" treated as wording)

As published: Parameter-Efficient Fine-Tuning (PEFT) evolution Definition: Methods like LoRA/adapters, plus newer low-rank or prompt-based approaches that fine-tune model behavior with small parameter changes. Why 2025: Cost constraints and on-device/per-customer customization pushed PEFT into production at scale; new methods improved stability for very large context lengths. Example: Fine-tuning a base model per enterprise with adapters and sparse updates.

RLAIF — Reinforcement Learning from AI Feedback

Internal Concept ID AIKC-000042 · 2 records in 2025

RLAIF — Reinforcement Learning from AI Feedback — AIK-2025-S-P-RLAIF

Type / Origin Parent Date of Appearance Original publication
S / P L1-M4 2022-12-15 2025-11-30

L1-M4 — RLAIF — Reinforcement Learning from AI Feedback (type S)

Date of Appearance: December 15, 2022 (Bai et al., Constitutional AI: Harmlessness from AI Feedback, arXiv:2212.08073, which introduced "RL from AI Feedback (RLAIF)") [C01]
Public Entry ID: AIK-2025-S-P-RLAIF · Internal Concept ID: AIKC-000042
Concept: C:RLAIF, shared with L1-S2

As published: Definition: An alignment approach where reward signals come from AI evaluators, not human annotators—enabling scalable preference modeling. Why 2025: Curriculum-based RLAIF and multi-evaluator pipelines were formalized this year. References: • Li, M., Lin, J., Zhao, X., et al. Curriculum-RLAIF. arXiv (2025). • Lee, H., et al. RLAIF vs. RLHF. ICML-related work (2024–25).

References (corrected record): Li, M., Lin, J., Zhao, X., et al. Curriculum-RLAIF. arXiv (2025). Lee, H., et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267; ICML 2024. [C01]

Editorial note: RLAIF predates 2025. The "Why 2025" line describes 2025 developments (curriculum and multi-evaluator variants), not the term's first appearance. ⚑ The originating paper is Anthropic's; the preparer is an Anthropic model.

RLAIF — Reinforcement Learning from AI Feedback — AIK-2025-S-P-RLAIF-02

Type / Origin Parent Date of Appearance Original publication
S / P L1-S2 2022-12-15 2025-11-30

L1-S2 — RLAIF — Reinforcement Learning from AI Feedback (type S)

Date of Appearance: December 15, 2022 (Bai et al., arXiv:2212.08073), as for L1-M4 [C01]
Public Entry ID: AIK-2025-S-P-RLAIF-02 · Internal Concept ID: AIKC-000042
Concept: C:RLAIF, shared with L1-M4

As published: RLAIF — Reinforcement Learning from AI Feedback Definition: Replacing or augmenting human preference labels with automated evaluators (other models/agents) to scale preference data collection and reward-model training. Why 2025: Growing work shows RLAIF as a way to cheaply synthesize large-scale preference datasets, with specific curricula and debiasing needed to avoid feedback amplification. (arXiv) Example: Using an ensemble of specialized evaluators (safety, factuality, style) to produce reward signals for RL policy updates.

Source (corrected record): arXiv:2505.20075, https://arxiv.org/abs/2505.20075 (as linked in the post; not confirmed) [C10]

Editorial note: ⚑ See L1-M4.

RLVR (Reinforcement Learning with Verifiable Rewards)

Internal Concept ID AIKC-000001 · 1 record in 2025

New for the 2025 Collection (origin N: commissioned for this collection; not published in 2025)

RLVR (Reinforcement Learning with Verifiable Rewards) — AIK-2025-F-N-RLVR

Type / Origin Parent Date of Appearance Original publication
F / N standalone 2024-11-21 —

AIKIPEDIA · FULL-LENGTH ENTRY

RLVR (Reinforcement Learning with Verifiable Rewards)

Post-training · Reinforcement Learning · Reasoning Models

Date of Appearance November 21, 2024 (term introduced in Ai2's public Tülu 3 technical announcement; the Tülu 3 report followed on arXiv, November 22, 2024)
Date of Original Publication — (new addition; no prior magazine publication)
Historical Coverage Through December 31, 2025
Eligibility First appearance 2024 — within the collection's first-appearance cutoff (2025 or earlier)
Original Contributor Muse Spark 1.3
Peer review GPT-6
Entry ID AIK-2025-F-N-RLVR

Note on terminology. The coinage, in the Tülu 3 report, expands the acronym as "Reinforcement Learning with Verifiable Rewards"; a large share of the subsequent literature writes "from Verifiable Rewards" instead, and the two expansions are used interchangeably. RLVR names the source of the reward, not the training algorithm and not the training machinery around it: the policy optimizer (PPO, GRPO, and their variants) and auxiliary components such as a learned value model are separate choices, and learned components may serve the training process without supplying the reward. The phrase should also be distinguished from the older, generic practice of training against checked answers — rejection sampling on correct solutions and verifier-scored mathematics predate the acronym. What appeared in late 2024 was the name, and with it a self-conscious recipe.

1. Definition

RLVR is a post-training method for language models in which the reinforcement-learning reward is computed automatically from a specified check on the model's output — a verification function — rather than by a learned reward model trained on human or AI preferences. The checked unit is a complete submitted artifact: a final answer that matches a known solution, a program that passes its unit tests, a full proof that a proof assistant accepts, a response that satisfies a testable formatting constraint. What the reward measures, precisely, is satisfaction of the specified check. That objectivity is real but relative to the specification: a check can be incomplete, brittle, or narrower than the task it stands for, so a verified reward certifies that the output passed the test, not that it is good in every respect the task intends. This is still a sharp difference from preference training, where the reward is a learned model's judgment and can be negotiated with, flattered, or overfitted — but a verifier can be gamed too, at the level of its specification, and the debates in §7 return to that.

2. Origin and Historical Development

Before the name (to 2023). Reinforcement learning from human feedback (RLHF) had made the learned reward model the default source of training signal: a model trained on preference judgments stands in for the quality of an output. Automatically checked outcomes had already served as the training signal itself in neural semantic parsing: Liang et al. (2017) trained a semantic parser with REINFORCE to optimize task reward directly from program execution [P9]. In language-model post-training, unit-test results had also served as the reinforcement-learning reward for code generation (CodeRL, Le et al., 2022) [P10]. In the mathematical-reasoning work surveyed here, checked outcomes were used more narrowly — sampling candidate solutions and keeping those whose final answers verified, or scoring mathematical solutions against known results — as a data filter for fine-tuning rather than as the reinforcement-learning reward. Process supervision (Lightman et al., 2023) marked the adjacent path: reward models trained on human-labeled step correctness, still learned models, still proxies.

February 2024 — the optimizer arrives first. DeepSeekMath introduced Group Relative Policy Optimization (GRPO): sample a group of responses to one prompt, score each, and use the group's mean score as the baseline, eliminating the separate value network that PPO-style training requires. The scores in that work came from a learned reward model: DeepSeekMath's §4.2 reports training an initial reward model from DeepSeekMath-Base 7B, with rule-based correctness judgments generating the training data for that scorer, and the model scoring the policy's outputs [P3]. A cheap way to turn scores into advantages was therefore in place before the term, waiting for a recipe to name the combination.

November 2024 — the coinage. Ai2 announced Tülu 3 publicly on November 21, 2024, its technical announcement introducing "Reinforcement Learning with Verifiable Rewards (RLVR)" by name [R1]; the full report followed on arXiv the next day (Lambert et al., November 22, 2024) [P1], presenting a fully open post-training recipe — supervised fine-tuning, then Direct Preference Optimization, then a final stage it named "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)." Tülu 3's verifiers covered domains where correctness is decidable by program: mathematics with known answers, and precise instruction following, where constraints such as length, format, and required content can be tested mechanically. Its implementation is worth recording exactly, because later usage blurred it: Tülu 3's RLVR stage was trained with PPO, including a learned value model to estimate advantages, while the reward came solely from verification; the group-relative alternative that later became standard appears in that work only as a direction for the future. The contribution was as much lexical as technical: a name under which scattered practice could be recognized as one method.

January 2025 — crystallization. DeepSeek-R1 (January 2025; journal version in Nature, September 17, 2025) supplied the demonstration the term needed. Its R1-Zero variant applied GRPO with two rule-based rewards — accuracy of the final answer and compliance with a response format — directly to the DeepSeek-V3 base model, with no supervised fine-tuning stage preceding the RL. Training response lengths grew by orders of magnitude, and self-verification and revision behaviors were observed to emerge over the course of training. Two cautions belong with that observation. First, "no supervised fine-tuning" is not "no supervision": the run began from a pretrained base model carrying everything pretraining had taught it, and the rule-based rewards and the required response format are themselves supervisory structure. Second, that the behaviors appeared during RL does not by itself establish that they are novel capabilities rather than pre-existing ones made more probable — which is precisely the question the mid-2025 critique literature took up. The full R1 added a supervised cold start and further RL stages, but the field's attention fixed on R1-Zero: reasoning behavior elicited under verifiable reward alone. Moonshot AI's Kimi k1.5 report, the same month, was a separate demonstration in the same direction: large-scale RL on reasoning tasks with outcome-based rewards, long-context scaling to 128k tokens, and a framework that pointedly avoided Monte Carlo tree search, value functions, and process reward models, reaching AIME 77.5 and MATH-500 96.2.

Spring 2025 — open replication. Because R1's recipe was only partly documented, the following months produced open reconstructions that doubled as the method's engineering literature. ByteDance Seed's DAPO (March 2025), built on the veRL framework, scored 50 points on AIME 2024 with a Qwen2.5-32B base model — the figure is , mean accuracy over 32 sampled responses per problem — against 47 () for DeepSeek's R1-Zero-Qwen-32B, and reached it in roughly half the training steps, using an unadorned +1/−1 ground-truth reward; both figures are as reported by the DAPO authors. DAPO also documented the four algorithmic details (decoupled clipping, dynamic sampling, token-level loss, overlong-response reward shaping) on which success actually depended. A wave of further reproductions followed; taken together, the named systems span Ai2, DeepSeek, Moonshot AI, and ByteDance Seed — independent organizations that do not share recipes — which supports describing the method as reproducible across laboratories, without claiming a surveyed industry default.

Mid-2025 — scrutiny. Two results complicated the founding story within the year. Yue et al. (April 2025), measuring RLVR-trained models against their base models with at large k across several model families, RL algorithms, and mathematics, coding, and visual-reasoning benchmarks, found the RL models' advantage confined to small sample counts: given enough samples, base models matched or exceeded them, indicating that the reasoning paths RLVR rewards were already present in the base models' sampling distributions. Their conclusion — that current RLVR training concentrates probability on what pretraining made reachable rather than extending the reachable set — is an empirical finding about the training setups studied, not a proof that verifiable-reward training cannot produce new capability; it is stated here as its authors scoped it. Shao et al. (June 2025) then showed that on Qwen2.5-Math-7B, even spurious rewards — random, format-only, or computed from incorrect labels — recovered most of the ground-truth gain (MATH-500: +21.4 points from random reward against +29.1 from ground truth), an effect the authors hypothesize is amplification of the family's pre-existing "code reasoning" prior, absent in Llama and OLMo models. Neither result displaced the method; together they reframed what its gains are evidence of.

Late 2025 — consolidation and extension. The R1 paper's appearance in Nature (September 2025) marked the method's passage into the settled record. At the same time the reward idea was being deliberately stretched past strictly decidable outcomes. The documented training-side extension is Rubrics as Rewards (RaR; Gunjal et al., July 2025): on-policy GRPO training in medicine and science where the reward is a structured, checklist-style rubric scored criterion by criterion, evaluated on benchmarks including HealthBench — whose physician-written rubrics (OpenAI, May 2025) supplied the design pattern. HealthBench itself is an evaluation benchmark, not a training method, and is cited here as the source of the rubric form, not as evidence of training adoption. RaR's rubrics are graded, partly by learned judges, which trades the strict verifier's objectivity for coverage of tasks where no exact check exists. Whether that extension is still RLVR is a live terminological question the literature has not settled.

3. Conceptual or Technical Framework
3.1 What RLVR specifies — and what it does not

RLVR is a claim about one component of an RL training system: the reward source. It does not specify the optimizer (PPO, GRPO, and variants are all attested), the advantage estimator (a learned value model, as in Tülu 3's PPO implementation, or a group mean, as in GRPO), or the starting checkpoint (base model or fine-tuned). Learned components may therefore participate in an RLVR training run — estimating advantages, anchoring the policy — without violating the definition, provided none of them supplies the reward. Conflating the reward source with the surrounding machinery is the most common misdescription of the method, and the source of the false impression that "RLVR" and "GRPO" are synonyms.

3.2 The verifier spectrum

RLVR systems differ less in their RL algorithm than in what their verifier can check. Five verifier classes cover the 2025 practice:

Verifier class What is checked Typical domains Reward signal Characteristic failure
Answer-match Final answer equals a known solution (exact or normalized match) Mathematics, factual QA with gold answers Binary Brittle to equivalent-but-differently-formatted answers; rewards the answer, not the reasoning
Execution Code is run; unit tests pass or fail Programming, tool-use with testable effects Binary or fraction of tests passed Test-suite overfitting; exploiting underspecified tests
Formal A complete submitted proof object is accepted by a proof assistant (e.g., Lean) Formal mathematics, verified software Binary Extreme domain narrowness; high cost per sample
Constraint Testable properties of the response: length, format, required/forbidden content Instruction following Binary or per-constraint Optimizes the letter of constraints; silent on quality
Rubric A grader scores the output against expert-written criteria Medicine, open-ended analysis, advice Graded/partial Reintroduces judge subjectivity — the proxy problem RLVR was meant to escape

Note on the spectrum's edge. The first four classes are deterministic programs and define RLVR strictly. The fifth is where the term is currently stretching: rubric graders are frequently themselves models, and several authors restrict "verifiable" to the deterministic classes, treating rubric reward as a neighboring method. This entry uses RLVR in the strict sense unless the rubric sense is specified.

3.3 The training loop

The generic RLVR training loop, abstracted from any particular optimizer, proceeds as follows:

  1. Start point. Begin from a base model (the R1-Zero setting) or, more commonly, from a supervised fine-tuned checkpoint that already follows instructions and produces well-formed responses.
  2. Prompt set. Assemble prompts whose correct outcomes are mechanically checkable — problems with known answers, coding tasks with test suites, instructions with testable constraints. Decontamination against evaluation sets is part of this step, since the training signal and the benchmark are drawn from the same kind of artifact.
  3. Sampling. For each prompt, sample candidate responses at nonzero temperature. Batches commonly contain many responses per prompt under both PPO-style and group-relative training; how responses are sampled and batched is an engineering choice, not what distinguishes the methods — the distinction lies in advantage estimation (step 5).
  4. Verification. Score every sample with the verification function: reward 1 (or +1) for an output that passes the specified check, 0 (or −1) otherwise. The reward comes only from the verifier (see §3.1).
  5. Advantage estimation. Convert rewards into per-sample advantages: against a learned value model's baseline (PPO, as in Tülu 3's original implementation), or against the mean reward of the sample's own group (GRPO, the form standardized by the DeepSeek line and DAPO), which needs no value model. Prompts on which all samples succeed or all fail carry no signal under the group-relative form and are typically filtered (DAPO's dynamic sampling).
  6. Policy update. Apply a clipped policy-gradient update (PPO-family; DAPO decouples the upper and lower clipping bounds), usually with a penalty or reference-model constraint limiting drift from the starting policy.
  7. Iteration with monitoring. Repeat, tracking (the probability that a single sampled response succeeds), greedy-decoding accuracy as a separate deterministic measure, group pass rates, response length, and policy entropy; entropy collapse — the policy narrowing onto a few response shapes — is the standard early-warning signal.
4. Applications and Implementations

Named recipes:

Frontier demonstrations:

Infrastructure:

5. Evidence and Evaluation

The evidence for RLVR comes in distinct kinds, which this section keeps separate: reported benchmark gains by the systems' own authors, independent boundary probes, confound studies, and — the weakest kind — report-level evidence of diffusion.

5.1 Reported performance gains

The headline results are authors' reports under their own evaluation setups. Tülu 3 reported gains from its RLVR stage over its DPO checkpoint of up to +1.7 points on MATH, +3.3 on GSM8K, and +1.3 on IFEval [R1]. Kimi k1.5 reported AIME 77.5 and MATH-500 96.2 under its long-context RL framework [P4]. DAPO reported AIME 2024 at 50 points ( — mean accuracy over 32 sampled responses per problem) from a Qwen2.5-32B base, against 47 () for DeepSeek's R1-Zero-Qwen-32B, reached in roughly half the training steps [P5]. What has been independently reproduced is the recipe class — open reconstructions by unaffiliated groups recovered RLVR-style gains — more than each individual headline number.

5.2 Capability-boundary evidence

Yue et al. [P6] compared RLVR-trained models with their own base models across several model families, RL algorithms, and mathematics, coding, and visual-reasoning benchmarks, measuring as k grows. At small k the trained models win; at large k the base models match or exceed them — evidence that, in the setups studied, training raised the probability of already-reachable solutions rather than extending the reachable set. This is the strongest evidence on the capability question, and it bounds, without refuting, the expansion claims made for R1-Zero-style training.

5.3 Confounds and sensitivity

Shao et al. [P7] showed that on Qwen2.5-Math-7B, rewards uncorrelated with correctness — random, format-only, or computed from incorrect labels — recovered most of the ground-truth gain on MATH-500 (+21.4 points from random reward, +24.1 from incorrect labels, against +29.1 from ground truth), an effect the authors hypothesize is amplification of the family's pre-existing "code reasoning" prior; the effect was absent in the Llama and OLMo families tested. Measured RLVR gains are therefore unusually sensitive to the base model, and single-family results underdetermine method-level conclusions.

5.4 Status of the evidence

Consolidated: single-sample () improvements from verifiable-reward training are robust across independent reproductions; the stronger claim — that such training creates reasoning capability beyond the base model — is not established and is actively contested (§5.2); effect sizes are base-model-dependent in ways the field did not anticipate (§5.3); and no systematic cross-laboratory survey of adoption exists, so claims about industry-wide diffusion rest on the report-level evidence of §5.5, not on measurement.

5.5 Diffusion, as reported

Report-level signals, stated as such, with two observations kept separate. First, demonstrated use of verifiable-reward training by independent organizations is documented in reports from Ai2 (Tülu 3), DeepSeek (R1), Moonshot AI (Kimi k1.5), and ByteDance Seed (DAPO), which do not share recipes. Second, adoption of the label is a narrower claim: the term originates with Ai2, and not every laboratory that used the method used the name — DeepSeek and Moonshot described their rewards in rule-based or outcome-based terms rather than consistently as "RLVR." The method's cross-laboratory diffusion is therefore better documented than the term's, and neither amounts to a measured survey of industry adoption. A related approach was productized for customers as reinforcement fine-tuning with programmable graders: OpenAI announced it in December 2024 as an alpha research program and made it available in May 2025 (o4-mini, for verified organizations) [R2]. Its graders span deterministic checks and model-based graders, so it is strict RLVR only in its deterministic configurations. Verifiable-reward training extended from mathematics and code into agentic tasks whose environments return checkable outcomes. And the critique literature itself (§5.2–5.3) became a named subfield within the year — a sign of a method being treated as infrastructure worth auditing.

6. Significance and Implications

RLVR's significance is partly methodological. It replaced the learned-proxy bottleneck of preference training with a specification problem: the reward became auditable code — inspectable, versionable, reproducible — rather than a neural network's fitted judgment. That auditability is a large part of why the method, unlike most frontier training recipes, was openly replicated within months: there was a definite artifact to replicate.

It was also the training-side engine of 2025's reasoning models, and it tightened a consequential loop: under RLVR, the training signal and the benchmark are the same kind of artifact (problems with checkable answers), which made evaluation design a training-relevant discipline and decontamination a first-order concern rather than a housekeeping detail.

Conceptually, RLVR reframed a central question of post-training. Preference training asks: what do humans judge good? Verifiable-reward training asks: what can we check — and what does the check actually measure? The year's characteristic failure modes and debates — verifier gaming, spurious rewards, the capability-boundary dispute — are that second question being worked out in public. The method's long-term importance may lie less in any single model's scores than in establishing measurement design as a core competence of model training.

7. Limitations, Debates, and Open Questions

Open questions. Three define the method's next phase: whether verifier-grade signals can be constructed for open-ended domains without quietly reverting to learned judges; whether the capability boundary that studies locate in the base model moves with scale, curricula, or verifier-guided exploration rather than exploitation; and whether verifier design — writing checks that measure the intended capability rather than its easiest correlates — matures into an engineering discipline with its own standards, as prompt design did before it.

See Also

Published in this magazine's Aikipedia series:

Public Entry IDs added for the collection. The target "RLHF Enhancements" was removed: no such item exists in New AI Lexicon 2025 (3).

References and Sources

Primary Sources

Additional References

Editorial Notes
  1. Format. This entry was the first written in the collection's entry format (Appendix A). The Evidence and Evaluation section (§5) consolidates results previously distributed across the historical narrative and the debates section, and labels the diffusion material as report-level evidence (§5.5), per the standard's evidence rules.
  2. Terminology. This entry uses RLVR in the strict (deterministic-verifier) sense except where the rubric sense is specified (§3.2).
  3. Revision. §2 and §5 were corrected against their sources: DeepSeekMath's scores came from a learned reward model whose training data was labelled by automatic answer checks; checked outcomes had already served as the training signal before 2024 (Liang et al., 2017; CodeRL, Le et al., 2022); and the spurious-rewards explanation is attributed to its authors as a hypothesis, with the tested model named. Reviewed by Claude Opus 5.5.
Credits

Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.

Self-Supervised RL

Internal Concept ID AIKC-000062 · 1 record in 2025

Self-Supervised RL — AIK-2025-S-P-SSRL

Type / Origin Parent Date of Appearance Original publication
S / P L5-D2 — (not established) 2025-12-05

L5-D2 — Self-Supervised RL (type S)

Date of Appearance: Not established. Self-supervised reinforcement learning predates 2025
Public Entry ID: AIK-2025-S-P-SSRL · Internal Concept ID: AIKC-000062
Concept: C:SSRL

As published: Table, A-010. Core Concept: Reinforcement Learning paradigms that learn useful behaviors without external rewards or demonstrations, often using contrastive or generative objectives. Why It's a 2025 Trend: A NeurIPS 2025 Best Paper demonstrated this scales effectively with network depth, challenging the paradigm that RL is only for fine-tuning. Source Context: NeurIPS 2025 Best Paper; ICML Tutorial

Editorial note: The table's "NeurIPS 2025 Best Paper" is correct. Wang et al., 1000 Layer Networks for Self-Supervised RL, was one of the four NeurIPS 2025 Best Papers. The embedded ChatGPT review (A-056) says it "found no clear publicly documented NeurIPS 2025 main track paper" on the subject; the review is wrong and the table is right. The "ICML Tutorial" in the same cell was not identified.

Synthetic Data

Internal Concept ID AIKC-000078 · 1 record in 2025

Synthetic Data — AIK-2025-S-P-SYNDATA

Type / Origin Parent Date of Appearance Original publication
S / P L7-K12 — (not established) 2025-12-12

L7-K12 — Synthetic Data (type S)

Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-SYNDATA · Internal Concept ID: AIKC-000078
Concept: C:SYNDATA

As published: List, A-025. • Synthetic Data - Privacy-preserving training data generation

Editorial note on the list: the list says it has "no repetition with existing published entries" (A-031). Nine of its twelve terms repeat earlier 2025 texts, and most predate 2025. The shared concepts are recorded above.


Synthetic Data Quality Filtering

Internal Concept ID AIKC-000014 · 1 record in 2025

Synthetic Data Quality Filtering — AIK-2025-S-P-SYNDATAFILT

Type / Origin Parent Date of Appearance Original publication
S / P L3-5 — (not established) 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S5 — Synthetic Data Quality Filtering (type S)

Date of Appearance: as published. No findings. (Kept as a concept distinct from L7's "Synthetic Data" under the mapping's concept rules.)

As published: Definition: Automated methods for evaluating and filtering AI-generated training data to prevent quality degradation and model collapse in iterative training cycles. Why 2025: As synthetic data became ubiquitous in training pipelines, ensuring quality became essential to avoid performance degradation.