🌏 中文版
Video status: Videos included. Source details
This post covers CS336 Spring 2026 Lecture 16: Reinforcement Learning from Verifiable Rewards, taught by Tatsunori Hashimoto on May 20, 2026. Its primary source is the official lecture_16.pdf.
Lecture 15's RLHF depends on human or model preference, and its proxy reward can become distorted under optimization. RLVR narrows the domain to tasks with reliable verifiers: mathematical rules, unit tests, compilers, or environment outcomes. Rewards are harder to game through style and can be generated automatically at scale.
PPO has many moving parts for language models
PPO samples trajectories from the current policy, computes advantages through rewards and a value model, and updates with a clipped objective. Language-model implementations add a per-token KL penalty to keep the policy near a reference; task reward arrives at the final token while advantages assign credit earlier.
Rollouts, reward and value inference, generalized advantage estimation, inner minibatch loops, and several checkpoints make the system complex. On-policy training also requires continual generation from the latest policy rather than reusing one fixed dataset like SFT.
GRPO replaces the value model with group statistics
Group Relative Policy Optimization samples several responses for one prompt and normalizes their rewards by the group's mean and standard deviation. The normalized value becomes an advantage. It removes the value function while retaining PPO-like ratios, clipping, and KL, reducing code and memory.
The group mean can serve as a baseline, but division by sample standard deviation changes weighting and need not preserve an unbiased gradient. All-correct or all-wrong groups have little variance; difficulties are rescaled differently. Token- versus sequence-level normalization can also favor certain lengths. GRPO is a practical tradeoff, not mathematically identical cheaper PPO.
Verifiers improve reward without eliminating reward hacking
Exact answers, tests, and environments are clearer than preference models, but verifiers remain incomplete. Parsers have loopholes, tests miss edge cases, and agents may read forbidden files. Format rewards can teach visible <think> tags rather than stronger reasoning.
A verifier contract therefore needs sandboxes, hidden tests, timeouts, resource limits, forbidden information paths, and human audits. Track correctness, format, cost, and safety separately before aggregation.
DeepSeek-R1 combines cold start, RL, and distillation
R1-Zero applies reasoning RL directly to a base model with accuracy and format rewards. Full R1 begins with a small high-quality chain-of-thought SFT cold start, applies RLVR, and later adds general SFT and RLHF. Distillation uses R1 traces to train smaller Qwen-family models.
The pipeline shows why “capability came from RL” cannot describe only the final step. Base model, prompt distribution, cold-start data, verifier, sampling budget, and distillation all contribute. Follow-up analyses also connect some apparently emergent formatting and length behavior to objective bias.
Curriculum and test-time compute share an axis
Kimi-like approaches resample by success rate, reducing solved questions and increasing those with learning signal, while curricula move from easy to hard. Sampling only the hardest tasks can yield groups with all-zero reward and no relative signal.
After training, longer chains, more samples, and search increase test-time compute. Evaluation must report accuracy alongside generated tokens, samples, and wall-clock, or extra inference budget will be mistaken for a raw model improvement.
Agentic RL extends verification into environments
Coding agents use repository tests; browser or GUI agents use final environment state. Qwen-like agent pipelines also employ repository-level mid-training, specialized SFT, and environment construction before RL. Tools, environments, and trajectory logging are part of the training system.
The bottleneck moves from annotation to rollout throughput. Switching between training and serving engines, variable trajectory lengths, stale policies, environment resets, and failure recovery all reduce GPU utilization.
A reliable RLVR experiment
Fix base checkpoint, prompt set, verifier version, group size, temperature, and token budget. Plot train reward, held-out verifier score, response length, entropy, KL, and pass@k separately. Audit extremely high-reward or suddenly longer trajectories and use different hidden tests to detect verifier overfitting.
Lecture 16 does not claim that RL has solved reasoning. Restricting rewards to checkable domains creates a cleaner scaling path, while algorithmic bias and systems cost remain.
Material fidelity
This lecture has a Spring 2026 schedule entry and a complete official PDF. This guide follows its PPO, GRPO, R1/Kimi/Qwen cases, test-time scaling, and agentic RL sections.
Course video sources
The video below is the corresponding Spring 2026 lecture published by Stanford Online. Lecture number and video ID were checked against the official playlist on 2026-10-10.
Content check: verified against the video transcript (2026-10-10): The video is Lecture 16 (Post-Training: RLVR, Tatsu). The captions cover GRPO (including the standard-deviation normalization discussion and Dr. GRPO), advantage, KL, DeepSeek-R1/R1-Zero and the SFT cold start, distilling R1 into Qwen, Kimi, format rewards, reward hacking, curriculum and agentic RL (including Qwen's mid-training). The captions mention PPO only once, so the PPO section leans on the slides. Not checked line by line; no body changes needed.
Original videos: Stanford CS336 Language Modeling from Scratch | Spring 2026 | Lecture 16: Post-Training - RLVR
Course and recording entries:
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Checked video IDs and lecture numbers against the official playlist; they are correct, and video titles now use the original titles.
- 2026-10-10: Checked the video content against its transcript. Topic and main concepts appear in the captions; no body changes needed.
References
Loading...