Table of Contents
- Speculative decoding: draft, then verify
- Off-policy drift and on-policy distillation
- Long context is not only a larger limit
- Where inference-time scaling spends compute
- Speculative acceptance
- Speed model and bottlenecks
- Why off-policy data drifts
- On-policy distillation
- Online/offline RL coordinates
- Extending positions
- Long-context data curriculum
- Attention memory and KV cache
- Long-context evaluation taxonomy
- Families of inference-time scaling
- Adaptive compute allocation
- Verifier ceilings
- An integrated experiment
- Material gap
- References
🌏 中文版
The official CS224N Winter 2026 schedule places lecture 13 on February 17, 2026, but does not name a lecturer; this article therefore attributes it only to the course staff. The official Reasoning 2/2 deck covers speculative decoding, off-policy drift and on-policy distillation, long-context extension, and inference-time scaling.
Speculative decoding: draft, then verify
Autoregressive generation with a large model requires an expensive forward pass per token. Speculative decoding has a smaller draft model propose several tokens and a target model verify them in parallel. Its acceptance correction accounts for differences between draft and target distributions, preserving the target distribution when implemented correctly.
Speedup requires a cheap draft sufficiently aligned with the target. Frequent rejection can erase the gain. This is a systems optimization, not a method for improving answer correctness.
Off-policy drift and on-policy distillation
When a model trains on fixed data generated by another policy, its deployed generation distribution can drift away from those training states. On-policy distillation samples states from the current student and asks a teacher for targets, matching training signals to trajectories the student actually visits.
This requires continuing sampling and teacher calls, increasing data-generation cost and risking stable replication of teacher errors.
Long context is not only a larger limit
Context extension involves position representations, training-length distributions, attention memory, and whether a model can use distant information. Position methods such as RoPE (RoFormer) can be scaled or retrained, but accepting input and reliably retrieving and reasoning over it are different tests.
Evaluation should separate locating information, integrating across passages, and robustness to distractors. A needle-in-a-haystack string lookup alone is not long-document reasoning.
Where inference-time scaling spends compute
Test-time compute scaling can add sampling, self-consistency, search, verifiers, or longer deliberation. Its value depends on task difficulty and verifier quality. Allocating resources after estimating difficulty can be more efficient than generating the same token budget for every problem.
Report quality with actual cost—tokens, latency, model calls, and verifier work. Otherwise “stronger reasoning” may simply mean unbounded compute.
Speculative acceptance
Draft several tokens, verify them in one target pass, and accept with a probability correction; naive argmax agreement changes the target distribution.
Speed model and bottlenecks
Measure draft cost, acceptance, verification, cache, hardware, batch, and sequence. Adaptive draft length handles high-entropy regions.
Why off-policy data drifts
As students update, they visit prefixes absent from fixed teacher trajectories. Offline coverage and on-policy freshness trade reuse against generation cost.
On-policy distillation
Generate from the current student and query the teacher on visited states. Target uncertain states to save cost; teacher errors still require outcome evaluation.
Online/offline RL coordinates
Online versus offline and on-policy versus off-policy describe different data relationships. Replay and correction create hybrids.
Extending positions
RoPE scaling, interpolation, and continued training trade short and long behavior. A mathematically available position does not ensure learned use.
Long-context data curriculum
Train on real or synthetic long dependencies, mask packed-document boundaries, retain short samples, and audit duplicate/leakage.
Attention memory and KV cache
Report prefill, decode throughput, and memory. FlashAttention, shared KV, windows, compression, and retrieval solve different bottlenecks.
Long-context evaluation taxonomy
Separate retrieval, aggregation, relational reasoning, global understanding, and distractor robustness. Needle tests cover only retrieval; compare RAG at equal cost.
Families of inference-time scaling
Sampling, voting, search, verifiers, refinement, and tools spend compute differently. Normalize tokens, FLOPs, or latency.
Adaptive compute allocation
Use calibrated difficulty or disagreement to allocate budgets, preserve a minimum, and report quality–cost frontiers.
Verifier ceilings
Oracle pass@N bounds selection. Verifiers can favor style or reject novel correct paths; open judging needs human audits.
An integrated experiment
Compare baseline, speculative speed, self-consistency, and verifier selection; stratify long-context and scaling gains by task type and difficulty.
Material gap
Winter 2026 recordings are not public. This article covers all four official agenda sections without reconstructing system details or comparisons absent from the deck.
References
Loading...