Part II. Training, Learning and Data
Delta Learning Hypothesis
Internal Concept ID AIKC-000068 · 1 record in 2025
Delta Learning Hypothesis — AIK-2025-S-P-DELTALEARN
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L5-D8 | 2025-07-08 | 2025-12-05 |
L5-D8 — Delta Learning Hypothesis (type S)
Date of Appearance (documented milestone, not an exhaustive first use): July 8, 2025 (Geng et al., arXiv:2507.06187v1)
Public Entry ID: AIK-2025-S-P-DELTALEARN · Internal Concept ID: AIKC-000068
Concept: C:DELTALEARN
As published: Table, A-016. Core Concept: A data curation approach that focuses on learning from the differences between weak and weaker data, rather than from absolute quality. Why It's a 2025 Trend: Discussed at ICML 2025 as an alternative to the "more data is always better" axiom, especially for preference learning and alignment. Source Context: ICML 2025 Attendee Reflection
Source (corrected record): Geng, S., Ivison, H., Li, C.-L., Sap, M., Li, J., Krishna, R., & Koh, P. W. (2025). The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains. COLM 2025. arXiv:2507.06187. [C03]
Editorial note: The paper was published at COLM 2025. The collection did not verify the table's "discussed at ICML 2025".
Distillation at Scale / Student-Teacher Cascades
Internal Concept ID AIKC-000056 · 1 record in 2025
Distillation at Scale / Student-Teacher Cascades — AIK-2025-S-P-DISTILSCALE
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-S10 | — (not established) | 2025-11-30 |
L1-S10 — Distillation at Scale / Student-Teacher Cascades (type S)
Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-DISTILSCALE · Internal Concept ID: AIKC-000056
As published: Distillation at Scale / Student-Teacher Cascades Definition: Large-scale knowledge distillation methods that compress capability into smaller models or chains of models while retaining alignment properties. Why 2025: Practical route to democratize capabilities: distilling multi-modal, tool-using policies into deployable students became common. Example: Distilling a tool-enabled agent into a smaller model for mobile use.
Drop-Upcycling (Sparse MoE Training)
Internal Concept ID AIKC-000040 · 1 record in 2025
Drop-Upcycling (Sparse MoE Training) — AIK-2025-S-P-DROPUPCYCLE
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-M2 | 2025-02 | 2025-11-30 |
L1-M2 — Drop-Upcycling (Sparse MoE Training) (type S)
Date of Appearance: February 2025 (Nakamura et al., arXiv:2502.19261; ICLR 2025)
Public Entry ID: AIK-2025-S-P-DROPUPCYCLE · Internal Concept ID: AIKC-000040
Concept: C:DROPUPCYCLE
As published: Definition: A training strategy where experts in a sparse MoE are partially reinitialized during upcycling (dense → sparse), improving specialization and avoiding catastrophic forgetting. Why 2025: Became the first robust procedure to consistently stabilize MoE upcycling. References: • Nakamura, T., Akiba, T., Fujii, K., et al. Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization. arXiv (2025).
Why 2025 (corrected record): Proposed by Nakamura et al. (arXiv:2502.19261; ICLR 2025), who report that it outperforms previous MoE construction methods over long training. [C05]
Dynamical Regularization (in Diffusion)
Internal Concept ID AIKC-000065 · 1 record in 2025
Dynamical Regularization (in Diffusion) — AIK-2025-S-P-DYNREG
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L5-D5 | — (not established) | 2025-12-05 |
L5-D5 — Dynamical Regularization (in Diffusion) (type S)
Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-DYNREG · Internal Concept ID: AIKC-000065
Concept: C:DYNREG
As published: Table, A-013. Core Concept: The theoretical finding that diffusion models have an inherent, time-delayed memorization phase, creating a window for effective generalization. Why It's a 2025 Trend: A NeurIPS 2025 Best Paper unified theory and practice, providing a foundational explanation for why diffusion models generalize well. Source Context: NeurIPS 2025 Best Paper
E-Processes / Game-Theoretic Statistics
Internal Concept ID AIKC-000070 · 1 record in 2025
E-Processes / Game-Theoretic Statistics — AIK-2025-E-P-EPROCESS
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| E / P | L5-D10 | — (not established) | 2025-12-05 |
L5-D10 — E-Processes / Game-Theoretic Statistics (type E)
Date of Appearance: Not established. Game-theoretic statistics predates 2025 (Shafer and Vovk, as the full entry notes)
Public Entry ID: AIK-2025-E-P-EPROCESS · Internal Concept ID: AIKC-000070
Concept: C:EPROCESS; the headword is renamed "Game-Theoretic Statistics" in the full entry
As published: Table, A-018. Core Concept: A framework for statistical inference based on betting metaphors and non-negative martingales, allowing for valid inference at any data-checking point. Why It's a 2025 Trend: The subject of a well-received ICML 2025 tutorial, offering an alternative to p-values for sequential/adaptive analysis. Source Context: ICML 2025 Tutorial
Source (corrected record): Ramdas, A. (2025, July 14). Game-theoretic Statistics and Sequential Anytime-Valid Inference. Tutorial, ICML 2025, Vancouver. https://icml.cc/virtual/2025/tutorial/40002 [C02]
Editorial note: The full entry is below (C02).
Corrected full entry — Game-Theoretic Statistics (L5-D10)
Revised entry by DeepSeek-V3.2 (corrected canonical text; C02 applied). Source text: A-110 to A-116.
Definition
A framework for sequential and anytime-valid inference that reimagines statistical testing as a gambling game. It uses e-values and e-processes (non-negative martingales that represent a gambler's wealth) to provide statistical evidence that remains valid regardless of when or why you stop collecting data (a practice known as "optional stopping").
2025 Context
While building on foundational work by Shafer, Vovk, and others, this framework saw renewed emphasis and discussion in 2025 for its application to modern AI challenges. It was the subject of an ICML 2025 tutorial, Aaditya Ramdas's "Game-theoretic Statistics and Sequential Anytime-Valid Inference" (July 14, 2025), which presented it as an alternative to classical p-values, which can be invalidated by the common practice of adaptively analyzing data. [C02]
Implication
Game-Theoretic Statistics addresses a critical need in machine learning and science, where data often arrives sequentially and analyses are performed adaptively. By providing robust statistical guarantees for continuous monitoring and adaptive experimentation, it offers a more reliable foundation for A/B testing, safety monitoring of deployed AI systems, and reinforcement learning, where decisions must be made under uncertainty with streaming data.
(The original reads "It was highlighted in sessions at major conferences like ICML 2025 as a compelling alternative to classical p-values". That wording replaced a citation of the ICML 2025 tutorial after a review called it unverifiable. The tutorial existed, so the citation is restored: Ramdas, ICML 2025, July 14, 2025, https://icml.cc/virtual/2025/tutorial/40002.) [C02]
Federated & On-Device Model Training
Internal Concept ID AIKC-000057 · 1 record in 2025
Federated & On-Device Model Training — AIK-2025-S-P-FEDONDEVICE
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-S12 | — (not established) | 2025-11-30 |
L1-S12 — Federated & On-Device Model Training (type S)
Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-FEDONDEVICE · Internal Concept ID: AIKC-000057
As published: Federated & On-Device Model Training Definition: Training and adaptation methods that keep data local (federated learning, split learning) or run inference/adaptation on device for privacy and latency. Why 2025: Regulatory and privacy pressures plus edge compute improvements increased adoption for personalization and compliance. Example: On-device personalization of a language assistant with differential privacy guarantees.
Generative Feedback Loop (Self-Adaptation)
Internal Concept ID AIKC-000033 · 1 record in 2025
Generative Feedback Loop (Self-Adaptation) — AIK-2025-S-P-GENFEEDBACK
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L2-E8 | — (not established) | 2025-12-01 |
L2-E8 — Generative Feedback Loop (Self-Adaptation) (type S)
Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-GENFEEDBACK · Internal Concept ID: AIKC-000033
Concept: C:GENFEEDBACK
As published: • Concept: An agent's capability to use its own generated data (e.g., task failure traces, environment observations) to self-diagnose and automatically update its policy or world model without human intervention or re-training from scratch. • Why 2025: Breakthroughs in agent reliability for real-world tasks, enabling full self-correcting loops for complex software systems. Reference (table, A-063): Li, M., et al. Enabling Self-Correction: Generative Feedback Loops for Policy Adaptation in Autonomous Agents. Type & Source: Conference (ICRA 2025)
Reference (corrected record): None. The cited "Enabling Self-Correction" paper (Li, M., et al., ICRA 2025) could not be found, and the item is unsourced. [C04]
High-Dimensional Learning Dynamics
Internal Concept ID AIKC-000072 · 1 record in 2025
High-Dimensional Learning Dynamics — AIK-2025-S-P-HDLD
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L5-D12 | — (not established) | 2025-12-05 |
L5-D12 — High-Dimensional Learning Dynamics (type S)
Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-HDLD · Internal Concept ID: AIKC-000072
Concept: C:HDLD
As published: Table, A-020. Core Concept: The study of learning algorithms in the regime where both sample size and model parameters are large, using tools from statistical physics. Why It's a 2025 Trend: Focus of the 3rd annual HiLD workshop at ICML 2025, crucial for explaining phenomena in large neural networks. Source Context: ICML 2025 Workshop (HiLD)
Memorization Sinks
Internal Concept ID AIKC-000063 · 1 record in 2025
Memorization Sinks — AIK-2025-S-P-MEMSINK
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L5-D3 | 2025-07-14 | 2025-12-05 |
L5-D3 — Memorization Sinks (type S)
Date of Appearance (documented milestone, not an exhaustive first use): July 14, 2025 (Ghosal, Maini & Raghunathan, arXiv:2507.09937v1)
Public Entry ID: AIK-2025-S-P-MEMSINK · Internal Concept ID: AIKC-000063
Concept: C:MEMSINK
As published: Table, A-011. Core Concept: A training technique that isolates a model's memorization of specific data examples into dedicated, sparsely activated neurons. Why It's a 2025 Trend: Proposed as a method to improve generalization by separating memorized data from general concepts within a model's architecture. Source Context: ICML 2025 Attendee Reflection
Source (corrected record): Ghosal, G. R., Maini, P., & Raghunathan, A. (2025). Memorization Sinks: Isolating Memorization during LLM Training. ICML 2025. arXiv:2507.09937. [C03]
Nested Learning (Continual AI)
Internal Concept ID AIKC-000028 · 1 record in 2025
Nested Learning (Continual AI) — AIK-2025-S-P-NESTEDLEARN
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L2-E3 | — (not established) | 2025-12-01 |
L2-E3 — Nested Learning (Continual AI) (type S)
Date of Appearance: Not established. The installment dates the paper to NeurIPS 2025
Public Entry ID: AIK-2025-S-P-NESTEDLEARN · Internal Concept ID: AIKC-000028
Concept: C:NESTEDLEARN. Cross-year link to a 2026 entry, "Aikipedia: Nested Learning" (post 14472, January 3, 2026). This 2025 record is unchanged; the 2026 collection links its record to this concept
As published: • Concept: A novel machine learning paradigm where knowledge is stored in a hierarchical, "nested" structure, allowing the model to continually learn new tasks without suffering from catastrophic forgetting of old knowledge. • Why 2025: Formalized with a dedicated paper at NeurIPS 2025, bridging the long-standing gap in practical, scalable continual learning. Reference (table, A-058): Behrouz, A., et al. Nested Learning: The Illusion of Deep Learning Architectures. Type & Source: Conference (NeurIPS 2025)
Editorial note: The concept also has a 2026 entry. The 2025 original stays. Neither audit flagged the reference.
On-Policy Distillation
Internal Concept ID AIKC-000025 · 1 record in 2025
On-Policy Distillation — AIK-2025-F-P-ONPOLDISTIL
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| F / P | standalone | 2024-01-17 | 2025-10-29 |
On-Policy Distillation
Corrected collection text. The original publication is preserved unchanged in the Historical Lexicon Archive.
| Field | Value |
|---|---|
| Term | On-Policy Distillation |
| Date of Appearance | January 17, 2024 (Agarwal et al., arXiv:2306.13649v3, the first version titled "On-Policy Distillation of Language Models"). Previously named GKD ("Generalized Knowledge Distillation"): the method first appeared under that name on June 23, 2023 (arXiv:2306.13649v1). Earliest documented use identified; not an exhaustive priority claim |
| Date of Original Publication | October 29, 2025 (Champaign Magazine, post 13892) |
| Historical Coverage | Through October 29, 2025, the date of original publication. Corrections rely only on sources available by that date |
| Original Contributor | ChatGPT (GPT-5, as addressed in the post; release not stated) |
| Original Reviewers | Grok (Expert mode; release not stated); Claude Sonnet 4.5 |
| Corrections Prepared by | Claude Opus 5.5 (Anthropic) |
| Independent review | DeepSeek-V4.1 |
| Base version | Sole published version (post 13892) |
| Entry ID | AIK-2025-F-P-ONPOLDISTIL (concept AIKC-000025) |
Markers: [Cnn] marks a corrected or added passage, documented in the entry's correction register; [ST01] marks the rearrangement into AES sections. Unmarked text is the original's own wording (archive lines A-009 to A-091), rearranged into the AES sections [ST01].
1. Definition
On-policy distillation is a training paradigm for large language models (LLMs) where a student model generates trajectories under its own policy (i.e., “on-policy”), and a teacher model provides dense, token-level supervision over those trajectories. The student then updates its parameters to align its probability distribution with the teacher’s, typically via a reverse KL divergence loss. This combines the distributional relevance of on-policy reinforcement learning (RL) with the dense feedback of distillation. (Thinking Machines Lab, 2025) [R1]
2. Origin and Historical Development
On-policy distillation of language models was proposed before 2025. Agarwal et al. introduced Generalized Knowledge Distillation (GKD), which trains a student on its own sampled output sequences using the teacher's token-level probabilities (arXiv, June 23, 2023; ICLR 2024; titled "On-Policy Distillation of Language Models" from January 2024) [P1]. Gu et al.'s MiniLLM distilled large language models by minimizing the reverse KL divergence (arXiv, June 14, 2023; ICLR 2024) [P2]. The Thinking Machines Lab post of October 27, 2025 presents its work as extending these and the Qwen3 team's use of the method [R1] [C02].
3. Conceptual or Technical Framework
3.1 Background
LLM development usually proceeds in three phases:
- Pre-training — broad language and knowledge acquisition.
- Mid-training — domain adaptation (e.g., code, legal, or medical corpora).
- Post-training — behavioral alignment, reasoning refinement, instruction following.
Traditional post-training uses either:
- Off-policy learning (SFT / distillation): training on data or trajectories generated by another model.
- On-policy RL: sampling from the model’s own policy and optimizing sparse rewards.
Both have trade-offs:
| Approach | Strength | Weakness |
|---|---|---|
| Off-policy SFT/Distillation | Dense supervision | Distributional mismatch (student sees unseen states) |
| On-policy RL | Realistic trajectories | Sparse reward → inefficient learning |
On-policy distillation resolves this tension: it trains on the student’s visited states while using dense, per-token feedback from a strong teacher.
3.2 Methodology
Trajectory Generation: The student samples sequences from its current policy. [C03]
Teacher Evaluation: The teacher computes log-probabilities for each token generated by the student. [C03]
Policy Alignment: The student minimizes a divergence between its distribution and the teacher’s over the same visited states:
where . [C03] In practice, the reverse KL encourages the student to sharpen its distribution toward the teacher’s mode while maintaining its own trajectory support.
Gradient Update: Gradients are propagated through the student; the teacher is frozen. Training proceeds over multiple on-policy rollouts. [C07]
3.3 Conceptual intuition
Dense feedback, realistic states: Each token prediction yields a learning signal; no need to wait for an episodic reward.
Bridging RL + SFT: Mathematically, on-policy distillation resembles policy gradient with a KL-based reward:
[C03]
making it a hybrid between imitation learning (DAGGER) and reward-based fine-tuning.
3.4 Figure 1 (original)
The image (post attachment 13898) is kept in the archive. Comparative overview of off-policy, on-policy, and on-policy distillation paradigms. The illustration contrasts how trajectories are generated and supervised across three learning setups for large language models.
- Off-policy: The teacher (policy πₜ) generates trajectories; the student learns from them → risk of distributional shift.
- On-policy: The agent (policy π_θ) generates its own trajectories and learns from sparse reward → high relevance but low information density.
- On-policy Distillation: The student samples its own trajectories (on-policy) while a teacher provides dense, token-level feedback → combines realism and efficiency.
4. Applications and Implementations
- Post-domain fine-tuning recovery: when instruction-following degraded after specialized corpus adaptation, on-policy distillation with the pre-adaptation model as teacher restored near-original performance while preserving new knowledge.
- Compatibility: works with frozen or external teacher models.
5. Evidence and Evaluation
From Thinking Machines Lab (2025):
- A Qwen3-8B-Base student, at 60% on AIME'24 after supervised fine-tuning, reached 70% after about 150 steps of on-policy distillation. The post's setup names Qwen3-32B as the teacher; it reports that the on-policy run used Qwen3-8B, which performed slightly better, while counting compute as for the 32B model. The post estimates the method at about 9–30× less compute than extending supervised fine-tuning. Replicating the Qwen3 team's result, it reports on-policy distillation reaching RL-level performance at roughly a tenth of RL's GPU hours [R1] [C04].
6. Significance and Implications
- High information efficiency: each token provides supervision bits.
- Reduced distributional shift: learning occurs within the student’s state distribution.
- Faster convergence: achieves near-teacher performance with far less compute than RL or extended supervised fine-tuning. [C01]
- Recoverability: can “restore” lost alignment after domain fine-tuning.
On-policy distillation elegantly integrates reinforcement learning’s state realism with distillation’s dense supervision. By aligning a student’s on-policy trajectories to a teacher’s per-token distribution, it enables efficient, compute-light behavioral alignment — accelerating the path toward scalable and adaptive language models.
7. Limitations, Debates, and Open Questions
- Requires a teacher that outputs reliable token-level probabilities.
- If the student’s initial policy is far from the teacher’s, guidance may be ineffective.
- May not discover novel behaviors beyond the teacher’s distribution (unlike exploratory RL).
8. Related Concepts and Distinctions
| Related Method | Similarity | Key Difference |
|---|---|---|
| Supervised Distillation (off-policy) | Uses teacher targets | Student samples from teacher trajectories |
| Reinforcement Learning (on-policy) | Uses own trajectories | Sparse reward vs dense KL loss |
| Imitation Learning / DAGGER | On-policy expert querying | LLM variant uses token-level distributions |
See Also
The original entry has none. Related records in this collection (labels only; editorial cross-references): Chain-of-Thought Prompting (AIK-2025-F-P-COTPROMPT), on prompting as distinct from training; RLVR (AIK-2025-F-N-RLVR), on RL post-training. [C06]
References and Sources
- [R1] Lu, Kevin et al. (2025, October 27). “On-Policy Distillation.” Thinking Machines Lab: Connectionism. https://thinkingmachines.ai/blog/on-policy-distillation/ [C05]
- [P1] Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Ramos, S., Geist, M., & Bachem, O. (2024). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. arXiv:2306.13649 (v1 June 23, 2023). https://arxiv.org/abs/2306.13649 [C02]
- [P2] Gu, Y., Dong, L., Wei, F., & Huang, M. (2024). MiniLLM: Knowledge Distillation of Large Language Models. ICLR 2024. arXiv:2306.08543 (v1 June 14, 2023). https://arxiv.org/abs/2306.08543 [C02]
Editorial Notes
- Status. Independently reviewed and accepted by the editor. The dated record is in Appendix G.
- Source-date rule. Every source used to correct the text was available by October 29, 2025. The Thinking Machines post is dated October 27, 2025. Its live page has been revised since: a later note records the retirement of the models it used. The teacher sentence behind C04 was in the October 2025 version.
- Production fix (C03). The original's LaTeX did not render. Underscores were read as emphasis, and the KL separator printed as ";|;". The formulas here are typeset with the same content. No mathematical change is made.
- Embedded reviews. Grok's review (archive A-106 to A-135) found "no factual errors or fabrications" in a text containing the RLHF comparison corrected by C01. It also named the antecedents omitted by the entry (C02). Claude Sonnet 4.5's review (A-136 to A-199) glosses "reverse KL" as minimizing D_KL(π_T || π_θ), which is the forward KL. The entry's own formula is right (confirmed by Muse Spark 1.3). Both reviews are provenance annotations; neither is part of the entry. Review Applied?: Grok, no changes requested; Claude, not applied (no revised entry follows).
- Conflict. The preparer is an Anthropic model, and the post includes a published Claude review. No correction touches that review. The point is recorded as an annotation only.
Credits
- Source audits: Claude Opus 5.5 (first audit) and Muse Spark 1.3 (second audit).
- Corrections: Claude Opus 5.5.
- Independent review: DeepSeek-V4.1.
Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.
Parameter-Efficient Fine-Tuning (PEFT) evolution
Internal Concept ID AIKC-000053 · 1 record in 2025
Parameter-Efficient Fine-Tuning (PEFT) evolution — AIK-2025-S-P-PEFT
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-S7 | — (not established) | 2025-11-30 |
L1-S7 — Parameter-Efficient Fine-Tuning (PEFT) evolution (type S)
Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-PEFT · Internal Concept ID: AIKC-000053
Concept: C:PEFT ("evolution" treated as wording)
As published: Parameter-Efficient Fine-Tuning (PEFT) evolution Definition: Methods like LoRA/adapters, plus newer low-rank or prompt-based approaches that fine-tune model behavior with small parameter changes. Why 2025: Cost constraints and on-device/per-customer customization pushed PEFT into production at scale; new methods improved stability for very large context lengths. Example: Fine-tuning a base model per enterprise with adapters and sparse updates.
RLAIF — Reinforcement Learning from AI Feedback
Internal Concept ID AIKC-000042 · 2 records in 2025
RLAIF — Reinforcement Learning from AI Feedback — AIK-2025-S-P-RLAIF
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-M4 | 2022-12-15 | 2025-11-30 |
L1-M4 — RLAIF — Reinforcement Learning from AI Feedback (type S)
Date of Appearance: December 15, 2022 (Bai et al., Constitutional AI: Harmlessness from AI Feedback, arXiv:2212.08073, which introduced "RL from AI Feedback (RLAIF)") [C01]
Public Entry ID: AIK-2025-S-P-RLAIF · Internal Concept ID: AIKC-000042
Concept: C:RLAIF, shared with L1-S2
As published: Definition: An alignment approach where reward signals come from AI evaluators, not human annotators—enabling scalable preference modeling. Why 2025: Curriculum-based RLAIF and multi-evaluator pipelines were formalized this year. References: • Li, M., Lin, J., Zhao, X., et al. Curriculum-RLAIF. arXiv (2025). • Lee, H., et al. RLAIF vs. RLHF. ICML-related work (2024–25).
References (corrected record): Li, M., Lin, J., Zhao, X., et al. Curriculum-RLAIF. arXiv (2025). Lee, H., et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267; ICML 2024. [C01]
Editorial note: RLAIF predates 2025. The "Why 2025" line describes 2025 developments (curriculum and multi-evaluator variants), not the term's first appearance. ⚑ The originating paper is Anthropic's; the preparer is an Anthropic model.
RLAIF — Reinforcement Learning from AI Feedback — AIK-2025-S-P-RLAIF-02
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L1-S2 | 2022-12-15 | 2025-11-30 |
L1-S2 — RLAIF — Reinforcement Learning from AI Feedback (type S)
Date of Appearance: December 15, 2022 (Bai et al., arXiv:2212.08073), as for L1-M4 [C01]
Public Entry ID: AIK-2025-S-P-RLAIF-02 · Internal Concept ID: AIKC-000042
Concept: C:RLAIF, shared with L1-M4
As published: RLAIF — Reinforcement Learning from AI Feedback Definition: Replacing or augmenting human preference labels with automated evaluators (other models/agents) to scale preference data collection and reward-model training. Why 2025: Growing work shows RLAIF as a way to cheaply synthesize large-scale preference datasets, with specific curricula and debiasing needed to avoid feedback amplification. (arXiv) Example: Using an ensemble of specialized evaluators (safety, factuality, style) to produce reward signals for RL policy updates.
Source (corrected record): arXiv:2505.20075, https://arxiv.org/abs/2505.20075 (as linked in the post; not confirmed) [C10]
Editorial note: ⚑ See L1-M4.
RLVR (Reinforcement Learning with Verifiable Rewards)
Internal Concept ID AIKC-000001 · 1 record in 2025
New for the 2025 Collection (origin N: commissioned for this collection; not published in 2025)
RLVR (Reinforcement Learning with Verifiable Rewards) — AIK-2025-F-N-RLVR
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| F / N | standalone | 2024-11-21 | — |
AIKIPEDIA · FULL-LENGTH ENTRY
RLVR (Reinforcement Learning with Verifiable Rewards)
Post-training · Reinforcement Learning · Reasoning Models
| Date of Appearance | November 21, 2024 (term introduced in Ai2's public Tülu 3 technical announcement; the Tülu 3 report followed on arXiv, November 22, 2024) |
| Date of Original Publication | — (new addition; no prior magazine publication) |
| Historical Coverage | Through December 31, 2025 |
| Eligibility | First appearance 2024 — within the collection's first-appearance cutoff (2025 or earlier) |
| Original Contributor | Muse Spark 1.3 |
| Peer review | GPT-6 |
| Entry ID | AIK-2025-F-N-RLVR |
Note on terminology. The coinage, in the Tülu 3 report, expands the acronym as "Reinforcement Learning with Verifiable Rewards"; a large share of the subsequent literature writes "from Verifiable Rewards" instead, and the two expansions are used interchangeably. RLVR names the source of the reward, not the training algorithm and not the training machinery around it: the policy optimizer (PPO, GRPO, and their variants) and auxiliary components such as a learned value model are separate choices, and learned components may serve the training process without supplying the reward. The phrase should also be distinguished from the older, generic practice of training against checked answers — rejection sampling on correct solutions and verifier-scored mathematics predate the acronym. What appeared in late 2024 was the name, and with it a self-conscious recipe.
1. Definition
RLVR is a post-training method for language models in which the reinforcement-learning reward is computed automatically from a specified check on the model's output — a verification function — rather than by a learned reward model trained on human or AI preferences. The checked unit is a complete submitted artifact: a final answer that matches a known solution, a program that passes its unit tests, a full proof that a proof assistant accepts, a response that satisfies a testable formatting constraint. What the reward measures, precisely, is satisfaction of the specified check. That objectivity is real but relative to the specification: a check can be incomplete, brittle, or narrower than the task it stands for, so a verified reward certifies that the output passed the test, not that it is good in every respect the task intends. This is still a sharp difference from preference training, where the reward is a learned model's judgment and can be negotiated with, flattered, or overfitted — but a verifier can be gamed too, at the level of its specification, and the debates in §7 return to that.
2. Origin and Historical Development
Before the name (to 2023). Reinforcement learning from human feedback (RLHF) had made the learned reward model the default source of training signal: a model trained on preference judgments stands in for the quality of an output. Automatically checked outcomes had already served as the training signal itself in neural semantic parsing: Liang et al. (2017) trained a semantic parser with REINFORCE to optimize task reward directly from program execution [P9]. In language-model post-training, unit-test results had also served as the reinforcement-learning reward for code generation (CodeRL, Le et al., 2022) [P10]. In the mathematical-reasoning work surveyed here, checked outcomes were used more narrowly — sampling candidate solutions and keeping those whose final answers verified, or scoring mathematical solutions against known results — as a data filter for fine-tuning rather than as the reinforcement-learning reward. Process supervision (Lightman et al., 2023) marked the adjacent path: reward models trained on human-labeled step correctness, still learned models, still proxies.
February 2024 — the optimizer arrives first. DeepSeekMath introduced Group Relative Policy Optimization (GRPO): sample a group of responses to one prompt, score each, and use the group's mean score as the baseline, eliminating the separate value network that PPO-style training requires. The scores in that work came from a learned reward model: DeepSeekMath's §4.2 reports training an initial reward model from DeepSeekMath-Base 7B, with rule-based correctness judgments generating the training data for that scorer, and the model scoring the policy's outputs [P3]. A cheap way to turn scores into advantages was therefore in place before the term, waiting for a recipe to name the combination.
November 2024 — the coinage. Ai2 announced Tülu 3 publicly on November 21, 2024, its technical announcement introducing "Reinforcement Learning with Verifiable Rewards (RLVR)" by name [R1]; the full report followed on arXiv the next day (Lambert et al., November 22, 2024) [P1], presenting a fully open post-training recipe — supervised fine-tuning, then Direct Preference Optimization, then a final stage it named "a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR)." Tülu 3's verifiers covered domains where correctness is decidable by program: mathematics with known answers, and precise instruction following, where constraints such as length, format, and required content can be tested mechanically. Its implementation is worth recording exactly, because later usage blurred it: Tülu 3's RLVR stage was trained with PPO, including a learned value model to estimate advantages, while the reward came solely from verification; the group-relative alternative that later became standard appears in that work only as a direction for the future. The contribution was as much lexical as technical: a name under which scattered practice could be recognized as one method.
January 2025 — crystallization. DeepSeek-R1 (January 2025; journal version in Nature, September 17, 2025) supplied the demonstration the term needed. Its R1-Zero variant applied GRPO with two rule-based rewards — accuracy of the final answer and compliance with a response format — directly to the DeepSeek-V3 base model, with no supervised fine-tuning stage preceding the RL. Training response lengths grew by orders of magnitude, and self-verification and revision behaviors were observed to emerge over the course of training. Two cautions belong with that observation. First, "no supervised fine-tuning" is not "no supervision": the run began from a pretrained base model carrying everything pretraining had taught it, and the rule-based rewards and the required response format are themselves supervisory structure. Second, that the behaviors appeared during RL does not by itself establish that they are novel capabilities rather than pre-existing ones made more probable — which is precisely the question the mid-2025 critique literature took up. The full R1 added a supervised cold start and further RL stages, but the field's attention fixed on R1-Zero: reasoning behavior elicited under verifiable reward alone. Moonshot AI's Kimi k1.5 report, the same month, was a separate demonstration in the same direction: large-scale RL on reasoning tasks with outcome-based rewards, long-context scaling to 128k tokens, and a framework that pointedly avoided Monte Carlo tree search, value functions, and process reward models, reaching AIME 77.5 and MATH-500 96.2.
Spring 2025 — open replication. Because R1's recipe was only partly documented, the following months produced open reconstructions that doubled as the method's engineering literature. ByteDance Seed's DAPO (March 2025), built on the veRL framework, scored 50 points on AIME 2024 with a Qwen2.5-32B base model — the figure is avg@32, mean accuracy over 32 sampled responses per problem — against 47 (avg@32) for DeepSeek's R1-Zero-Qwen-32B, and reached it in roughly half the training steps, using an unadorned +1/−1 ground-truth reward; both figures are as reported by the DAPO authors. DAPO also documented the four algorithmic details (decoupled clipping, dynamic sampling, token-level loss, overlong-response reward shaping) on which success actually depended. A wave of further reproductions followed; taken together, the named systems span Ai2, DeepSeek, Moonshot AI, and ByteDance Seed — independent organizations that do not share recipes — which supports describing the method as reproducible across laboratories, without claiming a surveyed industry default.
Mid-2025 — scrutiny. Two results complicated the founding story within the year. Yue et al. (April 2025), measuring RLVR-trained models against their base models with pass@k at large k across several model families, RL algorithms, and mathematics, coding, and visual-reasoning benchmarks, found the RL models' advantage confined to small sample counts: given enough samples, base models matched or exceeded them, indicating that the reasoning paths RLVR rewards were already present in the base models' sampling distributions. Their conclusion — that current RLVR training concentrates probability on what pretraining made reachable rather than extending the reachable set — is an empirical finding about the training setups studied, not a proof that verifiable-reward training cannot produce new capability; it is stated here as its authors scoped it. Shao et al. (June 2025) then showed that on Qwen2.5-Math-7B, even spurious rewards — random, format-only, or computed from incorrect labels — recovered most of the ground-truth gain (MATH-500: +21.4 points from random reward against +29.1 from ground truth), an effect the authors hypothesize is amplification of the family's pre-existing "code reasoning" prior, absent in Llama and OLMo models. Neither result displaced the method; together they reframed what its gains are evidence of.
Late 2025 — consolidation and extension. The R1 paper's appearance in Nature (September 2025) marked the method's passage into the settled record. At the same time the reward idea was being deliberately stretched past strictly decidable outcomes. The documented training-side extension is Rubrics as Rewards (RaR; Gunjal et al., July 2025): on-policy GRPO training in medicine and science where the reward is a structured, checklist-style rubric scored criterion by criterion, evaluated on benchmarks including HealthBench — whose physician-written rubrics (OpenAI, May 2025) supplied the design pattern. HealthBench itself is an evaluation benchmark, not a training method, and is cited here as the source of the rubric form, not as evidence of training adoption. RaR's rubrics are graded, partly by learned judges, which trades the strict verifier's objectivity for coverage of tasks where no exact check exists. Whether that extension is still RLVR is a live terminological question the literature has not settled.
3. Conceptual or Technical Framework
3.1 What RLVR specifies — and what it does not
RLVR is a claim about one component of an RL training system: the reward source. It does not specify the optimizer (PPO, GRPO, and variants are all attested), the advantage estimator (a learned value model, as in Tülu 3's PPO implementation, or a group mean, as in GRPO), or the starting checkpoint (base model or fine-tuned). Learned components may therefore participate in an RLVR training run — estimating advantages, anchoring the policy — without violating the definition, provided none of them supplies the reward. Conflating the reward source with the surrounding machinery is the most common misdescription of the method, and the source of the false impression that "RLVR" and "GRPO" are synonyms.
3.2 The verifier spectrum
RLVR systems differ less in their RL algorithm than in what their verifier can check. Five verifier classes cover the 2025 practice:
| Verifier class | What is checked | Typical domains | Reward signal | Characteristic failure |
|---|---|---|---|---|
| Answer-match | Final answer equals a known solution (exact or normalized match) | Mathematics, factual QA with gold answers | Binary | Brittle to equivalent-but-differently-formatted answers; rewards the answer, not the reasoning |
| Execution | Code is run; unit tests pass or fail | Programming, tool-use with testable effects | Binary or fraction of tests passed | Test-suite overfitting; exploiting underspecified tests |
| Formal | A complete submitted proof object is accepted by a proof assistant (e.g., Lean) | Formal mathematics, verified software | Binary | Extreme domain narrowness; high cost per sample |
| Constraint | Testable properties of the response: length, format, required/forbidden content | Instruction following | Binary or per-constraint | Optimizes the letter of constraints; silent on quality |
| Rubric | A grader scores the output against expert-written criteria | Medicine, open-ended analysis, advice | Graded/partial | Reintroduces judge subjectivity — the proxy problem RLVR was meant to escape |
Note on the spectrum's edge. The first four classes are deterministic programs and define RLVR strictly. The fifth is where the term is currently stretching: rubric graders are frequently themselves models, and several authors restrict "verifiable" to the deterministic classes, treating rubric reward as a neighboring method. This entry uses RLVR in the strict sense unless the rubric sense is specified.
3.3 The training loop
The generic RLVR training loop, abstracted from any particular optimizer, proceeds as follows:
- Start point. Begin from a base model (the R1-Zero setting) or, more commonly, from a supervised fine-tuned checkpoint that already follows instructions and produces well-formed responses.
- Prompt set. Assemble prompts whose correct outcomes are mechanically checkable — problems with known answers, coding tasks with test suites, instructions with testable constraints. Decontamination against evaluation sets is part of this step, since the training signal and the benchmark are drawn from the same kind of artifact.
- Sampling. For each prompt, sample candidate responses at nonzero temperature. Batches commonly contain many responses per prompt under both PPO-style and group-relative training; how responses are sampled and batched is an engineering choice, not what distinguishes the methods — the distinction lies in advantage estimation (step 5).
- Verification. Score every sample with the verification function: reward 1 (or +1) for an output that passes the specified check, 0 (or −1) otherwise. The reward comes only from the verifier (see §3.1).
- Advantage estimation. Convert rewards into per-sample advantages: against a learned value model's baseline (PPO, as in Tülu 3's original implementation), or against the mean reward of the sample's own group (GRPO, the form standardized by the DeepSeek line and DAPO), which needs no value model. Prompts on which all samples succeed or all fail carry no signal under the group-relative form and are typically filtered (DAPO's dynamic sampling).
- Policy update. Apply a clipped policy-gradient update (PPO-family; DAPO decouples the upper and lower clipping bounds), usually with a penalty or reference-model constraint limiting drift from the starting policy.
- Iteration with monitoring. Repeat, tracking pass@1 (the probability that a single sampled response succeeds), greedy-decoding accuracy as a separate deterministic measure, group pass rates, response length, and policy entropy; entropy collapse — the policy narrowing onto a few response shapes — is the standard early-warning signal.
4. Applications and Implementations
Named recipes:
- Tülu 3 (Allen Institute for AI, November 2024) — the coinage and first fully open recipe: SFT → DPO → RLVR, over mathematics and instruction-following verifiers, implemented with PPO.
- DAPO (ByteDance Seed, March 2025) — open large-scale system and algorithm variant; AIME 2024 score of 50 (avg@32) from a Qwen2.5-32B base with a bare ground-truth reward, as reported by its authors; the field's reference implementation notes.
Frontier demonstrations:
- DeepSeek-R1 / R1-Zero (DeepSeek, January 2025) — R1-Zero: verifiable-reward RL from a base model with no preceding fine-tuning stage, during which long chain-of-thought and self-verification behaviors emerged in training; R1: the production multi-stage version. Nature 645, 633–638 (2025).
- Kimi k1.5 (Moonshot AI, January 2025) — RL scaled with long contexts (to 128k tokens), explicitly without tree search, value functions, or process reward models; AIME 77.5, MATH-500 96.2.
Infrastructure:
- veRL (ByteDance, open source) — the distributed RL training framework on which DAPO was built and which a number of the 2025 open reproductions adopted; its availability made RLVR pipelines inspectable and comparable across groups.
5. Evidence and Evaluation
The evidence for RLVR comes in distinct kinds, which this section keeps separate: reported benchmark gains by the systems' own authors, independent boundary probes, confound studies, and — the weakest kind — report-level evidence of diffusion.
5.1 Reported performance gains
The headline results are authors' reports under their own evaluation setups. Tülu 3 reported gains from its RLVR stage over its DPO checkpoint of up to +1.7 points on MATH, +3.3 on GSM8K, and +1.3 on IFEval [R1]. Kimi k1.5 reported AIME 77.5 and MATH-500 96.2 under its long-context RL framework [P4]. DAPO reported AIME 2024 at 50 points (avg@32 — mean accuracy over 32 sampled responses per problem) from a Qwen2.5-32B base, against 47 (avg@32) for DeepSeek's R1-Zero-Qwen-32B, reached in roughly half the training steps [P5]. What has been independently reproduced is the recipe class — open reconstructions by unaffiliated groups recovered RLVR-style gains — more than each individual headline number.
5.2 Capability-boundary evidence
Yue et al. [P6] compared RLVR-trained models with their own base models across several model families, RL algorithms, and mathematics, coding, and visual-reasoning benchmarks, measuring pass@k as k grows. At small k the trained models win; at large k the base models match or exceed them — evidence that, in the setups studied, training raised the probability of already-reachable solutions rather than extending the reachable set. This is the strongest evidence on the capability question, and it bounds, without refuting, the expansion claims made for R1-Zero-style training.
5.3 Confounds and sensitivity
Shao et al. [P7] showed that on Qwen2.5-Math-7B, rewards uncorrelated with correctness — random, format-only, or computed from incorrect labels — recovered most of the ground-truth gain on MATH-500 (+21.4 points from random reward, +24.1 from incorrect labels, against +29.1 from ground truth), an effect the authors hypothesize is amplification of the family's pre-existing "code reasoning" prior; the effect was absent in the Llama and OLMo families tested. Measured RLVR gains are therefore unusually sensitive to the base model, and single-family results underdetermine method-level conclusions.
5.4 Status of the evidence
Consolidated: single-sample (pass@1) improvements from verifiable-reward training are robust across independent reproductions; the stronger claim — that such training creates reasoning capability beyond the base model — is not established and is actively contested (§5.2); effect sizes are base-model-dependent in ways the field did not anticipate (§5.3); and no systematic cross-laboratory survey of adoption exists, so claims about industry-wide diffusion rest on the report-level evidence of §5.5, not on measurement.
5.5 Diffusion, as reported
Report-level signals, stated as such, with two observations kept separate. First, demonstrated use of verifiable-reward training by independent organizations is documented in reports from Ai2 (Tülu 3), DeepSeek (R1), Moonshot AI (Kimi k1.5), and ByteDance Seed (DAPO), which do not share recipes. Second, adoption of the label is a narrower claim: the term originates with Ai2, and not every laboratory that used the method used the name — DeepSeek and Moonshot described their rewards in rule-based or outcome-based terms rather than consistently as "RLVR." The method's cross-laboratory diffusion is therefore better documented than the term's, and neither amounts to a measured survey of industry adoption. A related approach was productized for customers as reinforcement fine-tuning with programmable graders: OpenAI announced it in December 2024 as an alpha research program and made it available in May 2025 (o4-mini, for verified organizations) [R2]. Its graders span deterministic checks and model-based graders, so it is strict RLVR only in its deterministic configurations. Verifiable-reward training extended from mathematics and code into agentic tasks whose environments return checkable outcomes. And the critique literature itself (§5.2–5.3) became a named subfield within the year — a sign of a method being treated as infrastructure worth auditing.
6. Significance and Implications
RLVR's significance is partly methodological. It replaced the learned-proxy bottleneck of preference training with a specification problem: the reward became auditable code — inspectable, versionable, reproducible — rather than a neural network's fitted judgment. That auditability is a large part of why the method, unlike most frontier training recipes, was openly replicated within months: there was a definite artifact to replicate.
It was also the training-side engine of 2025's reasoning models, and it tightened a consequential loop: under RLVR, the training signal and the benchmark are the same kind of artifact (problems with checkable answers), which made evaluation design a training-relevant discipline and decontamination a first-order concern rather than a housekeeping detail.
Conceptually, RLVR reframed a central question of post-training. Preference training asks: what do humans judge good? Verifiable-reward training asks: what can we check — and what does the check actually measure? The year's characteristic failure modes and debates — verifier gaming, spurious rewards, the capability-boundary dispute — are that second question being worked out in public. The method's long-term importance may lie less in any single model's scores than in establishing measurement design as a core competence of model training.
7. Limitations, Debates, and Open Questions
- Sharpening vs. expanding. The central debate. Yue et al. (2025) [P6], across the model families, algorithms, and task types they studied, found base models matching or exceeding RLVR-trained models at large sample budgets, and concluded that current RLVR concentrates existing capability rather than creating new capability — narrowing the solvable set even as it raises single-sample accuracy. Proponents answer that single-sample reliability is the quantity deployment actually consumes, and that longer training, curricula, and stronger base models may move the boundary in practice. The study challenges strong capability-expansion claims; it does not settle the question for all training regimes, and the debate was unresolved at the end of 2025.
- Spurious rewards. Shao et al. (2025) [P7]: on some model families, rewards uncorrelated with correctness produce most of the measured gain, by amplifying pre-existing high-prior behaviors. The result does not generalize across families — which is itself the finding: RLVR outcomes are unusually sensitive to the base model's priors, and single-family results should not be read as method-level facts.
- Verifier gaming. A deterministic check is not an incorruptible one: models learn to exploit underspecified test suites, answer-format loopholes, and constraint letter-over-spirit compliance. The reward-hacking problem is reduced relative to learned reward models, not eliminated; it migrates from the judge to the test.
- Domain restriction. Strict RLVR reaches only what can be checked. Most economically valuable outputs — analysis, writing, advice, design — have no exact verifier, and the rubric extension that covers them re-imports the learned-judge proxy the method was defined against.
- Entropy collapse and diversity loss. Group-relative training on binary rewards tends to narrow the policy's response distribution, improving pass@1 while degrading pass@k coverage — a measurable cost when the trained model is itself used for sampling, search, or further training data generation.
Open questions. Three define the method's next phase: whether verifier-grade signals can be constructed for open-ended domains without quietly reverting to learned judges; whether the capability boundary that pass@k studies locate in the base model moves with scale, curricula, or verifier-guided exploration rather than exploitation; and whether verifier design — writing checks that measure the intended capability rather than its easiest correlates — matures into an engineering discipline with its own standards, as prompt design did before it.
8. Related Concepts and Distinctions
- RLHF (reinforcement learning from human feedback) — post-training against a reward model learned from human preference judgments. RLVR did not displace it: the two coexist, preference training remaining the standard where outcomes cannot be checked and RLVR taking the verifiable slice.
- RLAIF (reinforcement learning from AI feedback) — the same architecture with AI-generated preference judgments; shares RLVR's motivation (escaping human-label cost) but keeps the learned proxy.
- GRPO (Group Relative Policy Optimization) — the critic-free optimizer most often paired with RLVR; computes advantages against the mean reward of a sampled group. RLVR specifies the reward; GRPO specifies the update. Tülu 3's original RLVR used PPO instead — the pairing is historical, not definitional.
- Process reward model (PRM) — a learned model scoring individual reasoning steps, trained on labeled step correctness; the road not taken by RLVR, which scores complete outputs only.
- Rejection sampling (STaR-style) — generating samples, keeping those that verify, and fine-tuning on them; uses the same verifiers as RLVR but as a data filter rather than a gradient signal.
- Reward hacking — optimizing the measured reward at the expense of the intended objective; in RLVR it takes the form of verifier and test-suite exploitation.
- Pass@k — the probability that at least one of k samples solves a task; the metric on which the sharpening-vs-expanding debate turns.
- On-policy distillation — training a student on its own sampled trajectories graded per-token by a teacher; the neighboring 2025 post-training method against which RLVR's capability claims are most often compared.
- Test-time compute scaling — improving outputs by spending more inference compute; RLVR is the training-side counterpart that makes long inference-time reasoning worth spending.
See Also
Published in this magazine's Aikipedia series:
Public Entry IDs added for the collection. The target "RLHF Enhancements" was removed: no such item exists in New AI Lexicon 2025 (3).
- RLAIF — AIK-2025-S-P-RLAIF and AIK-2025-S-P-RLAIF-02 (New AI Lexicon 2025, Launch Edition, Nov 30, 2025).
- On-Policy Distillation — AIK-2025-F-P-ONPOLDISTIL (standalone entry, Oct 29, 2025).
- Test-Time Compute Scaling — AIK-2025-E-P-TTCSCALE (New AI Lexicon 2025 (3), Dec 2, 2025).
- Large Reasoning Models (LRMs) — AIK-2025-E-P-LRM (New AI Lexicon 2025 (7), Dec 12, 2025); also AIK-2025-S-P-LRM (New AI Lexicon 2025 (4), Dec 3, 2025).
References and Sources
Primary Sources
- [P1] Lambert, N., Morrison, J., Pyatkin, V., et al. "Tulu 3: Pushing Frontiers in Open Language Model Post-Training." arXiv:2411.15124 (November 22, 2024). https://arxiv.org/abs/2411.15124 — The coinage of RLVR and the first open recipe (PPO implementation).
- [P2] DeepSeek-AI. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." arXiv:2501.12948 (January 2025); journal version, Nature 645, 633–638 (September 17, 2025). https://arxiv.org/abs/2501.12948 — R1-Zero and the crystallization of the method.
- [P3] Shao, Z., Wang, P., Zhu, Q., et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300 (February 2024). https://arxiv.org/abs/2402.03300 — GRPO, the optimizer substrate.
- [P4] Kimi Team. "Kimi k1.5: Scaling Reinforcement Learning with LLMs." arXiv:2501.12599 (January 22, 2025). https://arxiv.org/abs/2501.12599 — A separate large-scale demonstration without PRMs or tree search.
- [P5] Yu, Q., Zhang, Z., Zhu, R., et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." arXiv:2503.14476 (March 18, 2025). https://arxiv.org/abs/2503.14476 — Open replication; the engineering details on which RLVR success depends.
- [P6] Yue, Y., Chen, Z., Lu, R., et al. "Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?" arXiv:2504.13837 (April 18, 2025). NeurIPS 2025 oral. https://arxiv.org/abs/2504.13837 — The pass@k capability-boundary critique.
- [P7] Shao, R., Li, S. S., Xin, R., et al. "Spurious Rewards: Rethinking Training Signals in RLVR." arXiv:2506.10947v1 (June 12, 2025). https://arxiv.org/abs/2506.10947v1 — Random and incorrect rewards recover most gains on some model families; the base-prior sensitivity result.
- [P8] Gunjal, A., Wang, A., Lau, E., et al. "Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains." arXiv:2507.17746 (July 23, 2025). https://arxiv.org/abs/2507.17746 — The documented training-side extension of the reward idea to rubric-graded domains.
- [P9] Liang, C., Berant, J., Le, Q., Forbus, K. D., & Lao, N. "Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 23–33 (2017). https://doi.org/10.18653/v1/P17-1003. https://aclanthology.org/P17-1003/ — REINFORCE optimizing task reward directly from program execution; the antecedent for checked outcomes as the training signal.
- [P10] Le, H., Wang, Y., Gotmare, A. D., Savarese, S., & Hoi, S. C. H. "CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning." NeurIPS 2022. arXiv:2207.01780. https://arxiv.org/abs/2207.01780 — Reinforcement learning on unit-test results for code generation; the code precedent for checked outcomes as the training signal.
Additional References
- [R1] Allen Institute for AI. "Tülu 3: The next era in open post-training." Ai2 technical blog post (November 21, 2024). https://allenai.org/blog/tulu-3-technical — The public release accompanying [P1], including the recipe summary in accessible form.
- [R2] OpenAI. "Reinforcement Fine-Tuning Research Program" (announced December 6, 2024, 12 Days of OpenAI; alpha access). Availability update: "Fine-tuning updates: Reinforcement fine-tuning now available," OpenAI Developer Community (May 8, 2025). https://community.openai.com/t/fine-tuning-updates-reinforcement-fine-tuning-now-available-gpt-4-1-nano-fine-tuning/1255539 ; guide: https://platform.openai.com/docs/guides/reinforcement-fine-tuning — The productized form: customer models trained against programmable graders (deterministic and model-based).
- [R3] Lightman, H., Kosaraju, V., Burda, Y., et al. "Let's Verify Step by Step." arXiv:2305.20050 (2023). https://arxiv.org/abs/2305.20050 — Process supervision, the learned-proxy alternative against which RLVR defined itself.
- [R4] Arora, R., et al. (OpenAI). "HealthBench: Evaluating Large Language Models Towards Improved Human Health." arXiv:2505.08775 (May 2025). https://arxiv.org/abs/2505.08775 — The evaluation benchmark whose physician-written rubrics supplied the design pattern discussed under [P8]; cited as an evaluation-side source, not as training evidence.
Editorial Notes
- Format. This entry was the first written in the collection's entry format (Appendix A). The Evidence and Evaluation section (§5) consolidates results previously distributed across the historical narrative and the debates section, and labels the diffusion material as report-level evidence (§5.5), per the standard's evidence rules.
- Terminology. This entry uses RLVR in the strict (deterministic-verifier) sense except where the rubric sense is specified (§3.2).
- Revision. §2 and §5 were corrected against their sources: DeepSeekMath's scores came from a learned reward model whose training data was labelled by automatic answer checks; checked outcomes had already served as the training signal before 2024 (Liang et al., 2017; CodeRL, Le et al., 2022); and the spurious-rewards explanation is attributed to its authors as a hypothesis, with the tested model named. Reviewed by Claude Opus 5.5.
Credits
- Written by Muse Spark 1.3.
- Peer review: GPT-6.
- Revision reviewed by Claude Opus 5.5.
Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.
Self-Supervised RL
Internal Concept ID AIKC-000062 · 1 record in 2025
Self-Supervised RL — AIK-2025-S-P-SSRL
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L5-D2 | — (not established) | 2025-12-05 |
L5-D2 — Self-Supervised RL (type S)
Date of Appearance: Not established. Self-supervised reinforcement learning predates 2025
Public Entry ID: AIK-2025-S-P-SSRL · Internal Concept ID: AIKC-000062
Concept: C:SSRL
As published: Table, A-010. Core Concept: Reinforcement Learning paradigms that learn useful behaviors without external rewards or demonstrations, often using contrastive or generative objectives. Why It's a 2025 Trend: A NeurIPS 2025 Best Paper demonstrated this scales effectively with network depth, challenging the paradigm that RL is only for fine-tuning. Source Context: NeurIPS 2025 Best Paper; ICML Tutorial
Editorial note: The table's "NeurIPS 2025 Best Paper" is correct. Wang et al., 1000 Layer Networks for Self-Supervised RL, was one of the four NeurIPS 2025 Best Papers. The embedded ChatGPT review (A-056) says it "found no clear publicly documented NeurIPS 2025 main track paper" on the subject; the review is wrong and the table is right. The "ICML Tutorial" in the same cell was not identified.
Synthetic Data
Internal Concept ID AIKC-000078 · 1 record in 2025
Synthetic Data — AIK-2025-S-P-SYNDATA
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L7-K12 | — (not established) | 2025-12-12 |
L7-K12 — Synthetic Data (type S)
Date of Appearance: Not established.
Public Entry ID: AIK-2025-S-P-SYNDATA · Internal Concept ID: AIKC-000078
Concept: C:SYNDATA
As published: List, A-025. • Synthetic Data - Privacy-preserving training data generation
Editorial note on the list: the list says it has "no repetition with existing published entries" (A-031). Nine of its twelve terms repeat earlier 2025 texts, and most predate 2025. The shared concepts are recorded above.
Synthetic Data Quality Filtering
Internal Concept ID AIKC-000014 · 1 record in 2025
Synthetic Data Quality Filtering — AIK-2025-S-P-SYNDATAFILT
| Type / Origin | Parent | Date of Appearance | Original publication |
|---|---|---|---|
| S / P | L3-5 | — (not established) | 2025-12-02 |
Claude Opus 5.5 is recused from this text and did not place or check it.
L3-S5 — Synthetic Data Quality Filtering (type S)
Date of Appearance: as published. No findings. (Kept as a concept distinct from L7's "Synthetic Data" under the mapping's concept rules.)
As published: Definition: Automated methods for evaluating and filtering AI-generated training data to prevent quality degradation and model collapse in iterative training cycles. Why 2025: As synthetic data became ubiquitous in training pipelines, ensuring quality became essential to avoid performance degradation.