CS336 Lecture 16: RLVR Scales Reasoning with Verifiable Rewards, but GRPO Is Not Free PPO

TL;DR Lecture 16 moves from PPO to GRPO and RLVR. Math, code, and environment outcomes provide scalable rewards and avoid some preference-model overoptimization, but group-normalized advantages introduce difficulty and length bias while rollout infrastructure becomes the dominant cost.

#cs336 #rlvr #grpo #reasoning #reinforcement-learning

Table of Contents

PPO has many moving parts for language models
GRPO replaces the value model with group statistics
Verifiers improve reward without eliminating reward hacking
DeepSeek-R1 combines cold start, RL, and distillation
Curriculum and test-time compute share an axis
Agentic RL extends verification into environments
A reliable RLVR experiment
Material fidelity
References

Series: Reading Stanford CS336 (17 / 18)

CS336 Lecture 15: SFT Teaches Imitation; RLHF Begins Direct Preference Optimization CS336 Lecture 17: Multimodal Models Turn Images into Tokens, Then Reconcile Semantics with Detail

🌏 中文版

This post covers CS336 Spring 2026 Lecture 16: Reinforcement Learning from Verifiable Rewards, taught by Tatsunori Hashimoto on May 20, 2026. Its primary source is the official lecture_16.pdf.

Lecture 15's RLHF depends on human or model preference, and its proxy reward can become distorted under optimization. RLVR narrows the domain to tasks with reliable verifiers: mathematical rules, unit tests, compilers, or environment outcomes. Rewards are harder to game through style and can be generated automatically at scale.

PPO has many moving parts for language models

PPO samples trajectories from the current policy, computes advantages through rewards and a value model, and updates with a clipped objective. Language-model implementations add a per-token KL penalty to keep the policy near a reference; task reward arrives at the final token while advantages assign credit earlier.

Rollouts, reward and value inference, generalized advantage estimation, inner minibatch loops, and several checkpoints make the system complex. On-policy training also requires continual generation from the latest policy rather than reusing one fixed dataset like SFT.

GRPO replaces the value model with group statistics

Group Relative Policy Optimization samples several responses for one prompt and normalizes their rewards by the group's mean and standard deviation. The normalized value becomes an advantage. It removes the value function while retaining PPO-like ratios, clipping, and KL, reducing code and memory.

The group mean can serve as a baseline, but division by sample standard deviation changes weighting and need not preserve an unbiased gradient. All-correct or all-wrong groups have little variance; difficulties are rescaled differently. Token- versus sequence-level normalization can also favor certain lengths. GRPO is a practical tradeoff, not mathematically identical cheaper PPO.

Verifiers improve reward without eliminating reward hacking

Exact answers, tests, and environments are clearer than preference models, but verifiers remain incomplete. Parsers have loopholes, tests miss edge cases, and agents may read forbidden files. Format rewards can teach visible <think> tags rather than stronger reasoning.

A verifier contract therefore needs sandboxes, hidden tests, timeouts, resource limits, forbidden information paths, and human audits. Track correctness, format, cost, and safety separately before aggregation.

DeepSeek-R1 combines cold start, RL, and distillation

R1-Zero applies reasoning RL directly to a base model with accuracy and format rewards. Full R1 begins with a small high-quality chain-of-thought SFT cold start, applies RLVR, and later adds general SFT and RLHF. Distillation uses R1 traces to train smaller Qwen-family models.

The pipeline shows why “capability came from RL” cannot describe only the final step. Base model, prompt distribution, cold-start data, verifier, sampling budget, and distillation all contribute. Follow-up analyses also connect some apparently emergent formatting and length behavior to objective bias.

Kimi-like approaches resample by success rate, reducing solved questions and increasing those with learning signal, while curricula move from easy to hard. Sampling only the hardest tasks can yield groups with all-zero reward and no relative signal.

After training, longer chains, more samples, and search increase test-time compute. Evaluation must report accuracy alongside generated tokens, samples, and wall-clock, or extra inference budget will be mistaken for a raw model improvement.

Agentic RL extends verification into environments

Coding agents use repository tests; browser or GUI agents use final environment state. Qwen-like agent pipelines also employ repository-level mid-training, specialized SFT, and environment construction before RL. Tools, environments, and trajectory logging are part of the training system.

The bottleneck moves from annotation to rollout throughput. Switching between training and serving engines, variable trajectory lengths, stale policies, environment resets, and failure recovery all reduce GPU utilization.

A reliable RLVR experiment

Fix base checkpoint, prompt set, verifier version, group size, temperature, and token budget. Plot train reward, held-out verifier score, response length, entropy, KL, and pass@k separately. Audit extremely high-reward or suddenly longer trajectories and use different hidden tests to detect verifier overfitting.

Lecture 16 does not claim that RL has solved reasoning. Restricting rewards to checkable domains creates a cleaner scaling path, while algorithmic bias and systems cost remain.

Material fidelity

This lecture has a Spring 2026 schedule entry and a complete official PDF. This guide follows its PPO, GRPO, R1/Kimi/Qwen cases, test-time scaling, and agentic RL sections.