Table of Contents
- Next-token prediction is not user assistance
- The three-stage RLHF pipeline
- What complexity DPO removes
- Human and AI feedback
- SFT data defines the assistant interface
- How preference data is generated
- What a reward model learns
- Roles inside PPO
- DPO intuition and objective
- Quality control for human and AI feedback
- A post-training failure taxonomy
- A small preference experiment
- Material gap and numbering note
- References
🌏 中文版
The official CS224N Winter 2026 schedule places lecture 8 on January 29, but does not name a lecturer; this article therefore attributes it only to the course staff. The official deck, Post-training (RLHF, SFT, DPO), covers instruction fine-tuning, RLHF, InstructGPT/ChatGPT, limitations of RL and reward modeling, DPO, and human preference data versus AI feedback.
Next-token prediction is not user assistance
Pretraining learns to continue text. Users expect a model to follow instructions, choose a useful form, and decline inappropriate requests. Instruction fine-tuning applies supervised learning to instruction–ideal-response examples, shifting a base model toward assistant interaction.
Its coverage is bounded by demonstrations. Outside them, the model must generalize from pretraining, and demonstrators' styles and judgments become model preferences.
The three-stage RLHF pipeline
The typical RLHF pipeline in the InstructGPT paper starts with supervised fine-tuning, collects preference rankings over multiple responses to the same prompt, trains a reward model on those comparisons, and then uses an algorithm such as PPO to raise predicted reward while constraining the policy against a reference model.
A reward model does not measure “true good.” It approximates choices made by annotators under a given interface and rubric. A policy may exploit its gaps. RL optimization also brings expensive sampling, value-function fitting, and hyperparameter sensitivity.
What complexity DPO removes
Direct Preference Optimization trains a policy directly on chosen/rejected pairs. It rewrites reward differences as log-probability differences between the policy and reference policy, yielding a classification-style loss without separately fitting a reward model and running PPO.
Removing the RL stage does not remove the preference problem. DPO still depends on pair quality, a reference model, and regularization strength, and learns only preferences represented in data. Collapsing annotator disagreement into one winner hides the disagreement itself.
Human and AI feedback
Human feedback is expensive and slow and needs a clear rubric. AI feedback scales quickly but can amplify a judge model's biases and blind spots. Evaluation should separately report data provenance, annotator agreement, and automated judging rather than only a final win rate.
SFT data defines the assistant interface
Instruction examples teach content, format, length, refusal, and conversational style. Balance mixtures by task and token contribution, deduplicate, and evaluate both target behavior and retained base capabilities.
How preference data is generated
Candidates, sampling temperature, and model mixture set comparison difficulty. Use multidimensional rubrics, position randomization, repeated labels, and agreement. Disagreement may express plural values rather than noise.
What a reward model learns
[ P(y_w\succ y_l\mid x)=\sigma(r_\phi(x,y_w)-r_\phi(x,y_l)). ]
Pairwise training identifies differences, not calibrated absolute reward. Test length, style, and confidence shortcuts plus out-of-distribution candidates.
Roles inside PPO
The policy generates, the reward model scores, a reference supplies KL control, and a value function reduces variance. Log reward, KL, length, entropy, and independent evaluation; high predicted reward can still mean hacking.
DPO intuition and objective
DPO trains the policy-reference log-ratio difference for winners versus losers with a logistic loss, eliminating a separate reward model and online PPO. Beta, sequence normalization, masking, and shared prefixes matter. Offline simplicity does not solve coverage shift.
Quality control for human and AI feedback
Humans add context but cost more; AI judges scale but share model blind spots and style preferences. Route ambiguous and high-risk pairs to humans, maintain a human audit set, and version judge models like datasets.
A post-training failure taxonomy
Track capability regression, over-refusal, sycophancy, reward hacking, mode collapse, and calibration loss on explicit slices. Compare base, SFT, and preference-optimized checkpoints under one evaluation.
A small preference experiment
Create paired answers for twenty prompts, collect two independent human rankings, and compare an automated judge. Swap positions, control length, and alter confidence while holding correctness fixed to expose shortcuts.
Material gap and numbering note
Winter 2026 recordings are not public. The deck cover retains a stale “Lecture 7: Post-training” label, but the official schedule, date, and filename establish it as regular lecture 8. This article covers all six agenda items without inventing spoken examples.
References
Loading...