Part III. Reasoning and Inference

Chain-of-Thought Prompting

Internal Concept ID AIKC-000007 · 1 record in 2025

Chain-of-Thought Prompting — AIK-2025-F-P-COTPROMPT

Type / Origin Parent Date of Appearance Original publication
F / P standalone 2022-01-28 2025-10-30

AIKIPEDIA · FULL-LENGTH ENTRY · CORRECTED ORIGINAL PUBLICATION

Chain-of-Thought Prompting

Prompting Methods · Reasoning · Evaluation

Date of Appearance January 28, 2022 (Wei et al., arXiv:2201.11903v1; the earliest documented use of the term identified)
Date of Original Publication October 30, 2025 (Champaign Magazine, post 13946)
Historical Coverage Through October 30, 2025, the date of original publication. Corrections rely only on sources available by that date
Original Contributor Gemini 2.5 Pro (writer). Much of the final version's wording comes from ChatGPT's review (see Entry Provenance)
Original Reviewers Grok (release not stated in the post); ChatGPT (release not stated in the post)
Corrections Prepared by Claude Opus 5.5 (Anthropic)
Independent review Muse Spark 1.3; DeepSeek-V4.1
Base version Version 3, which the post declares final (one passage of Version 2 restored at C17)
Entry ID AIK-2025-F-P-COTPROMPT (Internal Concept ID AIKC-000007)

How to read this text. This is the corrected collection text of an original 2025 publication. The original, in all three of its published versions, is preserved unchanged in the Historical Lexicon Archive. A bracketed number such as [C07] marks a passage changed from the original; the correction register gives the original wording, the revision and the evidence for each. [E01] marks an editorial addition. Unmarked text is the original's wording, rearranged into AES sections.

1. Definition

Chain-of-Thought (CoT) prompting is a prompt-engineering technique for large language models that elicits intermediate reasoning steps (a "chain of thought") before producing a final answer.

2. Origin and Historical Development

CoT was introduced and empirically validated in 2022 (Wei et al., 2022) [P1] and quickly became a standard tool for improving multi-step reasoning on benchmarks such as GSM8K, MATH, and multi-hop commonsense problems.

The variants described in §3.2, in order of introduction [C14]: Few-Shot CoT (Wei et al., 2022) [P1] · Zero-Shot CoT (Kojima et al., 2022) [P2] · Tree-of-Thoughts (ToT, 2023) [P4] · Graph-of-Thoughts (GoT, 2023) [P5] [C01] · Trace-of-Thought prompting (2024) [P7] [C04] · Chain-of-Draft (CoD, 2025 preprint) [P6] [C02].

3. Conceptual or Technical Framework
3.1 Core concept and mechanism

Standard prompting typically involves providing a model with a question (and sometimes a few examples) and expecting an immediate, direct answer.

Standard prompt example:

Q: A juggler has 3 red balls. He buys 2 bags of blue balls. Each bag has 4 blue balls. How many balls does he have in total? A: 11

In contrast, Chain-of-Thought prompting demonstrates or elicits the process of reasoning.

Chain-of-Thought prompt example (few-shot):

Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now? A: Roger started with 5 balls. 2 cans of 3 balls each is 2 * 3 = 6 balls. So, 5 + 6 = 11 balls. The answer is 11.

Q: A juggler has 3 red balls. He buys 2 bags of blue balls. Each bag has 4 blue balls. How many balls does he have in total? A: The juggler starts with 3 red balls. He buys 2 bags of blue balls. Each bag has 4 blue balls, so 2 * 4 = 8 blue balls. The total number of balls is 3 + 8 = 11 balls. The answer is 11.

By providing the intermediate steps, the model learns to "think" through the problem. This approach is believed to work because it aligns the generative process of the model with a structured, logical sequence, similar to how a human would work through a problem. CoT aligns model generation with stepwise logical structure and is especially helpful for problems that require sequential decomposition.

3.2 Key variants

Foundational variants

Evolved and complex variants (2023 onward) [C01]

4. Applications and Implementations

Chain-of-Thought prompting and its derivatives are used in [C18]:

When to use CoT (practical)

5. Evidence and Evaluation

Improved performance [C07]. CoT dramatically improves LLM performance on tasks that were previously difficult, especially arithmetic word problems (e.g., GSM8K, MATH) [P1] [R2]. Its original evaluation also reported gains on commonsense and symbolic reasoning tasks (e.g., CSQA) [P1]. A later analysis across 14 models found the gains concentrated in math and symbolic reasoning. On commonsense and broad-knowledge benchmarks such as MMLU the gains were much smaller, and on MMLU they came mainly from questions that involve calculation [R2].

Scalability. The benefits of CoT prompting have been shown to scale with model size; larger models tend to produce more coherent and accurate reasoning chains [P1].

Diminishing returns (nuanced) [C06]. Several empirical studies and technical reports since 2023–2025 indicate that the marginal benefit of explicit CoT prompting decreases for some high-capability, recent LLMs and for certain tasks.

That said, this effect is task- and model-dependent. CoT remains most useful on mathematical and symbolic tasks [R2], and its variants remain valuable for long-horizon planning and complex combinatorial search [P4] [C06]. For non-reasoning models that do not reason step by step by default, explicit CoT prompting can still give small average gains [R4] [C06]. Practitioners should evaluate CoT cost/benefit per task and model.

Table 1. Selected author-reported results from the entry's primary sources [E01].

Method Setting Reported result Source
Zero-shot CoT InstructGPT (text-davinci-002), "Let's think step by step" vs. plain zero-shot MultiArith 17.7% → 78.7%; GSM8K 10.4% → 40.7% [P2]
Tree-of-Thoughts GPT-4, Game of 24 CoT prompting solved 4% of tasks; ToT 74% [P4]
Chain-of-Draft Several reasoning benchmarks Matched or surpassed CoT accuracy using as little as 7.6% of CoT's tokens [P6]
CoT on MMLU 14 models (meta-analysis and own evaluations) Up to 95% of CoT's MMLU gain comes from questions containing "=" [R2]

Table note: all figures are author-reported, under the authors' stated models, prompts and scoring. They are not independent replications [E01].

6. Significance and Implications

CoT quickly became a standard tool for improving multi-step reasoning (§2) [C14].

Reasoning elicited by prompting alone. Zero-shot CoT showed that a single instruction, with no examples, could improve reasoning across diverse tasks. Kojima et al. presented this as a hint that large models hold untapped zero-shot reasoning capabilities that simple prompting can extract [P2] [C17].

Interpretability. The generated chain of thought provides a readable trace of intermediate steps. This can help operators identify where a logical error occurred and audit the output. The trace is not guaranteed to reflect how the model actually reached its answer: CoT explanations can be plausible yet unfaithful [R1] (§7) [C08].

7. Limitations, Debates, and Open Questions

Limitations

Safety and interpretability. Although CoT provides a readable "reasoning trace," these sequences are model generations, not human-grade proofs. Chains may contain hallucinated or unsafe steps, and revealing internal chains can sometimes expose model-learned heuristics or biases. For high-stakes uses, combine CoT with external verification (symbolic checks, unit tests, or formal verification) and treat chains as hypotheses to be validated rather than ground truth.

See Also

General concepts with no entry in the 2025 collection [C16]: Prompt Engineering · In-Context Learning · Large Language Model (LLM) · Self-Consistency (AI) · Tree-of-Thoughts · Graph-of-Thoughts · Active Prompting.

Related records in the Aikipedia 2025 collection (editorial cross-references, not part of the original) [C16]:

References and Sources

Primary Sources [C10]

Additional References

Editorial Notes
  1. Status of this text. Independently reviewed and accepted by the editor. The historical publication is not replaced; it stays in the Historical Lexicon Archive. The dated record is in Appendix G.
  2. Base version. The original post publishes three versions. This text is based on the third, which the post declares final. One passage of the second version is restored in §6 [C17], with its wording aligned to the source's hedge.
  3. Source-date rule. Every source used to correct the historical body was available by October 30, 2025, the original publication date; the latest is [R4], June 8, 2025. No later development has been added.
  4. Qualified claim. The original said CoT "remains valuable … when working with smaller or mid-sized models." It gave no source. Its own primary source found no gain, or harm, below about 100 billion parameters for the 2022 models it tested [P1]. A 2025 study [R4] found small average gains for non-reasoning models that do not reason step by step by default. That study grouped models by reasoning behavior, not size. The claim is therefore restated in the form the sources support, and the size wording is not retained [C06]. The editor adopted this wording.
  5. Proposed substitutions. [R2] and [R4] stand in for the original's unspecified "related technical reports" [C05]. They are the preparer's choice, not the original's. The editor accepted them.
  6. Worked example. The tennis-ball exemplar in §3.1 follows the exemplar in Wei et al.'s Figure 1 with light rewording. The juggler problem is the original contributor's own.
  7. Precursors. Later literature cites Nye et al. (2021, "scratchpads") alongside Wei et al. as an origin of the technique; see, for example, [R1]. That precursor is noted here but not added to the historical body.
  8. Editorial addition. Table 1 [E01] is not in the original. It quotes author-reported figures from the entry's own sources so that §5 states evaluation conditions, as AES §5 requires. The editor ruled to keep it, as a labeled addition.
  9. Affiliation note. Two authors of [R1] list Anthropic affiliations, and the corrections were prepared by an Anthropic model. That correction [C08] was therefore checked independently: Muse Spark 1.3 found it warranted, and DeepSeek-V4.1 confirmed it at the primary record. DeepSeek's check of the affiliations themselves was at register level only.
  10. Application list. The original called CoT "a standard technique in" the six areas listed in §4 but gave no source for any of them. On the editor's ruling, the list is kept and the phrase is changed to "used in" [C18]. The text therefore no longer asserts, without a source, that the practice is standard.
Credits

Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.

Chain-of-Verification (CoVe)

Internal Concept ID AIKC-000021 · 1 record in 2025

Chain-of-Verification (CoVe) — AIK-2025-S-P-COVE

Type / Origin Parent Date of Appearance Original publication
S / P L3-12 2023-09 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S12 — Chain-of-Verification (CoVe) (type S)

Date of Appearance: 2023 (corrected — C02; Dhuliawala et al., arXiv:2309.11495, September 2023)
Headword note: published as "Chain-of-Verification (CoV)"; the paper's abbreviation is CoVe, used here.

As published: Definition: Method where models generate initial responses, then systematically verify claims through self-questioning and evidence gathering to reduce hallucinations. Why 2025: Became a standard technique for improving factual accuracy in high-stakes applications like medical and legal domains.

Editorial note: first appearance is 2023 (ACL 2024 Findings); the correction concerns the stated date and abbreviation, not the term's inclusion.


Controllable Generation Frameworks

Internal Concept ID AIKC-000017 · 1 record in 2025

Controllable Generation Frameworks — AIK-2025-S-P-CTRLGEN

Type / Origin Parent Date of Appearance Original publication
S / P L3-8 — (not established) 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S8 — Controllable Generation Frameworks (type S)

Date of Appearance: as published. No findings.

As published: Definition: Structured approaches for fine-grained control over generative model outputs through semantic steering, attribute manipulation, and constraint satisfaction. Why 2025: Moved beyond simple prompting to enable precise creative control in image, video, and text generation.

KV-Cache Sharing (On-Device Inference)

Internal Concept ID AIKC-000027 · 1 record in 2025

KV-Cache Sharing (On-Device Inference) — AIK-2025-E-P-KVSHARE

Type / Origin Parent Date of Appearance Original publication
E / P L2-E2 — (not established) 2025-12-01

L2-E2 — KV-Cache Sharing (On-Device Inference) (type E)

Date of Appearance: Not established.
Public Entry ID: AIK-2025-E-P-KVSHARE · Internal Concept ID: AIKC-000027
Concept: C:KVSHARE

As published: • Concept: A critical optimization technique that allows multiple models or tasks on a device (like a smartphone) to access the same Key-Value (KV) cache in memory, significantly reducing the memory and latency overhead for simultaneous LLM operations. • Why 2025: Became a core architectural innovation for enabling powerful, multi-functional on-device foundation models (like those detailed in the 2025 Apple Intelligence report). Reference (table, A-057): Ma, B., et al. SafeKV: Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference. Type & Source: Preprint (arXiv 2025)

Why 2025 (corrected record): The 2025 Apple Intelligence report describes KV-cache sharing in Apple's ~3B-parameter on-device model, where the second of two blocks reuses the KV cache of the first, cutting KV-cache memory and time-to-first-token by about 37.5% (Apple, Apple Intelligence Foundation Language Models: Tech Report 2025, arXiv:2507.13575, July 2025). That is sharing within one model, not across tasks or models. [C06]

Reference (corrected record): Chu, K., et al. (2025). SafeKV: Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference. arXiv:2508.08438. [C07]

Editorial note: The full entry is corrected below (C06, C07, C08, C12).

Corrected full entry — KV-Cache Sharing (On-Device Inference) (L2-E2)

Aikipedia Entry — 2025 Edition (corrected canonical text; C06, C07, C08 and C12 applied). Source text: A-134 to A-160.

Definition

KV-Cache Sharing is a resource optimization technique used primarily in the deployment of Large Language Models (LLMs) on multi-tenant or memory-constrained edge hardware (like personal devices). It allows the intermediate attention computation outputs, specifically the Key (K) and Value (V) vectors (the KV-Cache), to be shared or reused across multiple concurrent inference sessions, tasks, or users, drastically reducing memory bandwidth and latency.

The term is also used for reuse of the KV cache across layers or blocks within a single model, as in Apple's 2025 on-device model (see Historical Context). [C06]

Historical Context

The KV-Cache, essential for efficient auto-regressive generation, is the largest memory consumer during inference. Its size limited the context length and the number of users a server could support. Early sharing methods (pre-2025) focused on sharing common prefixes (like the system prompt) on large server farms.

In 2025, Apple's ~3B-parameter on-device foundation model used a within-model form of KV-cache sharing: the model is divided into two blocks, and the second block reuses the KV cache generated by the first. Apple reports this cuts KV-cache memory use and time-to-first-token by about 37.5% (Apple, Tech Report 2025, arXiv:2507.13575). [C06]

(The original's "The 2025 breakthrough applied this to on-device (local) inference, particularly for devices running multiple, small LLM-powered services simultaneously …" is replaced. The cited Apple report describes sharing within one model, not across services, and no Qualcomm report was identified.) [C06]

Technical Description

The KV-Cache stores the K and V vectors for all previously generated tokens. In a shared system, the cache memory is managed by a centralized scheduler (often integrated into the GPU's memory manager):

(The original's memory-reduction formula, "Memory Reduction ≈ Σ_{i∈Shared} S_i·N_i·2·D / B", and the sentence defining its variables are removed. The formula has no source, and its terms do not yield a memory quantity: it omits the number of layers and the bytes per element, it counts sharers rather than the copies avoided, and its division by batch size is unexplained.) [C12]

Notable Systems / Papers (2025)

(The original's "Apple Intelligence / Qualcomm Technical Reports on On-Device LLM Optimization. (2025)" is replaced by the Apple report above. The Qualcomm half could not be identified.) [C06]

Applications
Limitations
Open Questions

Multi-Agent Debate Systems

Internal Concept ID AIKC-000015 · 1 record in 2025

Multi-Agent Debate Systems — AIK-2025-S-P-MADEBATE

Type / Origin Parent Date of Appearance Original publication
S / P L3-6 2023-05 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S6 — Multi-Agent Debate Systems (type S)

Date of Appearance: May 2023 (corrected — C03; Du et al., arXiv:2305.14325)

As published: Definition: Frameworks where multiple AI agents argue different positions or perspectives before reaching a conclusion, improving reasoning quality and reducing bias. Why 2025: Emerged as a practical technique for complex decision-making and fact-checking applications.

Editorial note: first appearance is 2023; 2025 marks wider adoption and formalization.

Prefill–Decode Disaggregation

Internal Concept ID AIKC-000023 · 1 record in 2025

Prefill–Decode Disaggregation — AIK-2025-F-P-PDDISAGG

Type / Origin Parent Date of Appearance Original publication
F / P standalone 2024-01-18 2025-11-09

Prefill–Decode Disaggregation

Corrected collection text. The original publication is preserved unchanged in the Historical Lexicon Archive.

Field Value
Term Prefill–Decode Disaggregation
Date of Appearance January 18, 2024 (Zhong et al., DistServe, arXiv:2401.09670v1, "Disaggregating Prefill and Decoding"). The technique appeared earlier as "phase splitting": Patel et al., Splitwise, arXiv:2311.18677v1, November 30, 2023. Earliest documented use identified; not an exhaustive priority claim
Date of Original Publication November 9, 2025 (Champaign Magazine, post 14033)
Historical Coverage Through November 9, 2025, the date of original publication. Corrections rely only on sources available by that date
Original Contributor Kimi K2 Thinking (initial draft, linked from the post, not reproduced in it); ChatGPT (revised and final versions)
Original Reviewers Grok Expert; Claude 4.5 Sonnet; Gemini 2.5 Pro (named in the post; no review texts are published)
Corrections Prepared by Claude Opus 5.5 (Anthropic)
Independent review GLM 5.3 Flash
Base version Sole published version on the post (post 14033). The off-post Kimi text linked from the post is a provenance annotation, not a version (rulings X9/X15)
Entry ID AIK-2025-F-P-PDDISAGG (concept AIKC-000023)

Markers: [Cnn] marks a corrected or added passage and [ST01] the rearrangement into AES sections, documented in the entry's correction register. Unmarked text is the original's own wording (archive lines A-007 to A-083).

1. Definition

Prefill–decode disaggregation optimizes large language model (LLM) inference by separating the prefill phase (parallel prompt processing) from the decode phase (sequential token generation), enabling tailored hardware allocation and independent scaling for each phase. [ST01]

2. Origin and Historical Development

First proposed in Splitwise (2023), which splits prompt computation and token generation onto separate machines [8], and formalized as goodput-optimized disaggregated serving in DistServe (2024) [1], this approach has evolved rapidly with systems like Mooncake, FlowKV, and NVIDIA Dynamo addressing KV cache transfers, scheduling, and production-scale deployment. [C01] [ST01]

History / Development Timeline [C01] [C06]
Year Event
Late 2023 Splitwise (Microsoft; arXiv:2311.18677, ISCA 2024) splits prompt and token-generation phases onto separate machines.
2024 (OSDI) DistServe formalizes disaggregation; first goodput gains observed.
Late 2024 vLLM/Ray prototypes emerge.
Early 2025 (FAST ’25) Mooncake published; KV-centric orchestration demonstrated.
March 2025 (GTC) NVIDIA Dynamo launched; vendor-grade distributed inference framework.
April 2025 FlowKV arXiv release; low-latency KV transfer and scheduling.
Mid-2025 Dynamo v0.4 introduces Blackwell optimizations.
3. Conceptual or Technical Framework
3.1 Overview

LLM inference involves:

Disaggregation assigns these phases to separate clusters:

Clusters communicate via high-speed interconnects (e.g., NVLink, InfiniBand) to transfer KV cache states efficiently. Utilization differences are typical: prefill >90%, decode 20–40% GPU utilization, as reported in technical analyses of disaggregated serving [9]. [C02]

Monolithic serving mixes prefill and decode, causing idle hardware. [ST01]

3.2 KV-Cache Transfer

Efficient KV cache transfer is critical:

3.3 Scheduling and Load Balancing

Dynamic routing and admission control (e.g., NVIDIA Dynamo, vLLM experimental disaggregated prefill) help maintain throughput and fairness across workloads.

3.4 Figure 1 (original)

The image (post attachment 14035, file copilot_20251109_162736.png) is kept in the archive; its file name indicates it was generated with Microsoft Copilot. The post gives it no credit line. [C08] Architecture diagram:

  1. Prefill Clusters: Input prompt processed via parallel compute.
  2. KV Cache Stream: Optimized KV transfer (compressed tensors) moves data between prefill and decode.
  3. Decode Clusters: Sequential token generation using KV cache.
  4. Measurements: TTFT and ITL monitored at each phase.
4. Applications and Implementations
5. Evidence and Evaluation

Evaluations show workload-dependent gains—reduced latency, higher goodput—but benefits require high-bandwidth networks and careful orchestration. [ST01]

Phase-specific batching improves Time to First Token (TTFT) and Inter-Token Latency (ITL): [C04]

Cost efficiency varies: Splitwise reports 1.4× higher throughput at 20% lower cost, or 2.35× more throughput under the same cost and power budgets [8]; secondary technical analyses cite 15–40% reductions in infrastructure cost [9]. [C05]

Metrics include TTFT, ITL, throughput, P99 latency, tested on H100/A800 clusters using LongBench v1/v2 for long-context evaluation.

6. Significance and Implications

Disaggregation enables independent scaling, maximizing compute utilization for prefill and memory throughput for decode. Bursty workloads benefit most. [ST01]

7. Limitations, Debates, and Open Questions

Disaggregation relies on high-bandwidth fabrics. Slow networks can erode gains, particularly in heterogeneous setups. Hybrid or SLO-aware strategies are suggested for smaller models or low-bandwidth conditions.

Disaggregation is less effective for:

See Also

Cross-reference markings (editorial, not part of the original): targets in the collection carry their Public Entry IDs; the others are marked as having no entry.

References and Sources
  1. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving – arXiv:2401.09670, OSDI 2024.
  2. FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling – arXiv:2504.03775, 2025.
  3. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot (Qin et al.) – USENIX FAST 2025. https://www.usenix.org/system/files/fast25-qin.pdf [C07]
  4. NVIDIA Dynamo: A Low-Latency Distributed Inference Framework for Scaling Reasoning AI Models – NVIDIA Blog, March 2025.
  5. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding – GitHub (https://github.com/THUDM/LongBench).
  6. vLLM Documentation: Disaggregated Prefilling (Experimental) – vLLM Docs (https://docs.vllm.ai/).
  7. Disaggregated Inference: 18 Months Later – Hao AI Lab retrospective (https://hao-ai-lab.github.io/blogs/distserve-retro/).
  8. Splitwise: Efficient Generative LLM Inference Using Phase Splitting (Patel et al.) – arXiv:2311.18677 (November 30, 2023), ISCA 2024. [C01]
  9. Disaggregation in Large Language Models: the Next Evolution in AI Infrastructure – InfoQ article (September 29, 2025). https://www.infoq.com/articles/llms-evolution-ai-infrastructure/ [C02]
Editorial Notes
  1. Status. Independently reviewed and accepted by the editor. The dated record is in Appendix G.
  2. Source-date rule. Every source used to correct the text was available by November 9, 2025: Splitwise (November 30, 2023), the DistServe OSDI version (June 2024), FlowKV (April 2025), the Mooncake FAST '25 paper (February 2025), and the InfoQ article (September 29, 2025).
  3. Provenance. The post links the initial Kimi K2 Thinking draft (A-084); the link is not carried into the entry. The figure's file name indicates Microsoft Copilot as its generator; the post credits no one for it (C08). No review texts are published in the post, so Review Applied? cannot be assessed.
  4. Figure legend. Item 2 of the legend ("compressed tensors", A-064) describes KV transfer generically and is left as published. It is not a claim about FlowKV, whose mechanism is corrected at C03.
  5. Conflict. The preparer is an Anthropic model, and the post names Claude 4.5 Sonnet as a peer reviewer. No Claude text appears in the post.
Credits

Accepted by the editor, W.H.L. The dated record of this entry (versions, audits, reviews and rulings) is in Appendix G.

Quantization & Ultra-Low-Precision Inference

Internal Concept ID AIKC-000054 · 1 record in 2025

Quantization & Ultra-Low-Precision Inference — AIK-2025-S-P-QUANTINFER

Type / Origin Parent Date of Appearance Original publication
S / P L1-S8 — (not established) 2025-11-30

L1-S8 — Quantization & Ultra-Low-Precision Inference (type S)

Date of Appearance: Not established. The installment frames the term as a 2025 term.
Public Entry ID: AIK-2025-S-P-QUANTINFER · Internal Concept ID: AIKC-000054
As published: Quantization & Ultra-Low-Precision Inference Definition: Techniques (4-bit, 3-bit, mixed precision) and quant-aware training to run large models on commodity hardware without big accuracy loss. Why 2025: Widespread use in open-source and commercial models to reduce inference cost and enable edge deployment. Example: Post-training quantization + fine-tuning to run a 70B-equivalent model on a single server.

Speculative Decoding

Internal Concept ID AIKC-000016 · 1 record in 2025

Speculative Decoding — AIK-2025-S-P-SPECDECODE

Type / Origin Parent Date of Appearance Original publication
S / P L3-7 2022-11-30 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S7 — Speculative Decoding (type S)

Date of Appearance: November 30, 2022 (corrected — C01; Leviathan, Kalman & Matias, arXiv:2211.17192; Chen et al., arXiv:2302.01318, February 2023)

As published: Definition: Inference optimization technique where a small "draft" model generates candidate tokens that a larger model verifies in parallel, dramatically reducing latency. Why 2025: Became standard in production systems as latency requirements tightened for interactive applications.

Editorial note: the technique predates 2025; its 2025 relevance is production standardization, as the short form's own text describes.

Test-Time Compute Scaling

Internal Concept ID AIKC-000011 · 1 record in 2025

Test-Time Compute Scaling — AIK-2025-E-P-TTCSCALE

Type / Origin Parent Date of Appearance Original publication
E / P L3-2 2024-08-06 2025-12-02

Claude Opus 5.5 is recused from this text and did not place or check it.

L3-S2 — Test-Time Compute Scaling (type E)

Date of Appearance: 2024–2025 (crystallization; OpenAI o1, September 2024 — as the entry itself records). No appearance correction.

As published: Definition: Methods that improve model performance by allocating more computational resources during inference (e.g., longer reasoning chains, more sampling attempts) rather than only during training. Why 2025: Became a key alternative to pre-training scaling as models like o1 demonstrated dramatic improvements through inference-time computation.

Note: the full entry is corrected at C06 (restoration — Mathematical Framework equations restored from the published formulas) — see below.

Corrected full entry — Test-Time Compute Scaling (L3-S2)

Aikipedia Entry — 2025 Edition (corrected canonical text; correction C06 applied, marked [C06])

Definition

Test-Time Compute Scaling refers to methods that improve AI model performance by allocating additional computational resources during inference rather than only during training. Instead of generating responses immediately, models invest extra time in iterative reasoning, self-correction, verification, and exploration of multiple solution paths before producing final outputs.

Historical Context

Traditional language models operated on a simple paradigm: generate outputs as quickly as possible using fixed inference budgets. Pre-training scaling laws dominated the field from 2018–2024, with performance improvements achieved primarily by training larger models on more data.

In September 2024, OpenAI released the o1 model series, which demonstrated that performance consistently improves with more time spent thinking during inference. This marked a fundamental shift: as the field encountered data bottlenecks and diminishing returns from pre-training scaling laws, test-time compute emerged as a key driver of performance improvements.

By early 2025, the paradigm had crystallized with models like DeepSeek-R1 exploring reinforcement learning approaches to achieve strong reasoning through deep, step-by-step thinking. Multiple research groups published frameworks for achieving test-time scaling, making it accessible beyond closed proprietary systems.

Technical Description

Test-time compute encompasses several interconnected approaches: 1. Extended Chain-of-Thought (CoT) Reasoning

Models generate explicit reasoning traces before producing answers, allowing them to work through complex multi-step problems systematically.

2. Iterative Self-Refinement

The model generates an initial response, critiques it, identifies errors, and produces improved versions through multiple refinement cycles.

3. Best-of-N Sampling with Verification

4. Tree Search Strategies

Models explore multiple reasoning paths simultaneously, backtrack when needed, and search for optimal solutions through the space of possible reasoning chains.

5. Budget Forcing

A technique that controls test-time compute by forcefully terminating or lengthening the model's thinking process by appending "Wait" tokens multiple times, leading the model to double-check its answer and often fix incorrect reasoning steps.

Mathematical Framework

For a model policy π and input x, traditional inference produces:

y = π(x)

Test-time scaling introduces compute budget C:

y = argmaxy ∈ 𝒴(x, C) R(y)

where 𝒴(x, C) is the set of candidate responses generated using compute budget C, and R is a reward/verification function. [C06 — equations restored from the published formulas (the formula images' alt texts in the stored post)]

The performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute).

Notable Systems / Papers (2025)
Applications
Limitations
Open Questions