Table of Contents
- Decoding changes the model you observe
- From R1-Zero to R1 and distillation
- What PPO, GRPO, and DAPO change
- Define reasoning through failure conditions
- Greedy, beam, and sampling optimize different behavior
- Neural text degeneration
- R1-Zero's training signal
- The R1 multi-stage pipeline
- PPO's clipped update
- GRPO uses group-relative advantage
- DAPO's systems corrections
- Reward exploits
- Self-consistency and selection
- When more thought helps
- Behavioral tests of reasoning
- An executable experiment
- Material gap
- References
🌏 中文版
The official CS224N Winter 2026 schedule places lecture 12 on February 12, 2026, but does not name a lecturer; this article therefore attributes it only to the course staff. The official Reasoning 1/2 deck covers decoding, DeepSeek-R1, PPO/GRPO/DAPO, and the nature, success, and failure conditions of reasoning.
Decoding changes the model you observe
Greedy decoding selects the highest-probability token at every step; it is fast but myopic. Beam search retains several high-probability prefixes and suits tasks seeking a likely full sequence. In open generation, maximizing probability can become repetitive and bland. Sampling draws from the distribution, while temperature, top-k, and top-p control diversity.
A “model answer” is therefore the model distribution plus a decoding policy. A reasoning comparison that does not fix or report decoding may measure sampling rather than parameters.
From R1-Zero to R1 and distillation
The deck uses DeepSeek-R1 to discuss reinforcement learning on verifiable tasks. R1-Zero applies RL directly to a base model, producing long reasoning behavior but readability and language-mixing problems. R1 adds cold-start data and staged training. R1-distill teaches smaller models with reasoning data generated by stronger ones.
These names denote different data and training pipelines. Their results cannot all be summarized as “RL creates reasoning by itself.”
What PPO, GRPO, and DAPO change
PPO clips policy-ratio updates and commonly uses a value model, reference policy, and multiple loss terms. GRPO forms advantages from relative rewards among several answers to one problem, avoiding a separate critic. DAPO addresses clipping, dynamic sampling, length bias, and stability in large-scale RL systems.
Chain-of-Thought Prompting makes intermediate reasoning text an explicit method, but algorithm names are not conclusions. Outcomes also depend on verifiable rewards, task distributions, sample counts, and data filtering. If reward can be gamed, longer text may merely create more surface signals for the evaluator.
Define reasoning through failure conditions
Long text is not necessarily correct reasoning. Evaluation should inspect answer accuracy, step verification, consistency across samples, and robustness to rephrasing. More test-time compute can waste work on easy questions; repeated thought does not guarantee progress when knowledge is absent or reward cannot distinguish quality.
Greedy, beam, and sampling optimize different behavior
Greedy is local; beam needs length handling; temperature, top-k, and top-p alter diversity and risk. Fix sample count, token budget, stop rules, and selection when comparing reasoning.
Neural text degeneration
Measure repetition, diversity, entropy, and task quality together. Surface penalties can hide loops without repairing model or objective causes.
R1-Zero's training signal
Verifiable outcome rewards support scalable RL but do not validate intermediate steps and can be exploited. Readability and language mixing show correctness reward does not specify communication.
The R1 multi-stage pipeline
Cold-start SFT, reasoning RL, rejection sampling, and broader alignment use distinct data and objectives. Distillation needs novel-problem and no-rationale controls.
PPO's clipped update
PPO clips policy-ratio changes and uses value, KL, and token masks. Monitor reward, KL, entropy, clip fraction, and length.
GRPO uses group-relative advantage
Multiple answers to one prompt form relative advantages without a critic. Group size costs rollout compute; all-equal groups provide weak signal.
DAPO's systems corrections
Decoupled clipping, dynamic sampling, token losses, and overlong shaping jointly address stability. Inspect ablations and compute rather than attributing everything to one acronym.
Reward exploits
Test parser formats, invalid reasoning plus correct answers, long outputs, and code side effects. Process rewards improve credit assignment but introduce expensive, nonunique step labels.
Self-consistency and selection
Report pass@N, selected accuracy, and oracle coverage. Majority helps only when errors vary; verifier quality determines whether minority correct paths can be selected.
When more thought helps
Use adaptive compute for moderately difficult, verifiable tasks with relevant knowledge. Report quality–cost frontiers.
Behavioral tests of reasoning
Use paraphrases, distractors, counterfactuals, and equivalent shorter problems. Separate step validity, answers, consistency, and calibration from fluent rationales.
An executable experiment
Compare greedy, sampling, self-consistency, and verifier selection under equal token budgets; stratify gains by baseline difficulty and classify failures.
Preserve raw trajectories and verifier decisions so generation coverage and selector accuracy can be recomputed separately.
Material gap
Winter 2026 recordings are not public. This article covers all four agenda sections and does not generalize one R1 case study into a universal result for reasoning models.
References
Loading...