Table of Contents
- SFT data teaches content and style together
- Mid-training moves instruction-like data into pretraining
- Why optimize preferences after imitation
- PPO learns a reward and updates a policy
- DPO writes preference into a direct classification loss
- Reward overoptimization and mode collapse
- An auditable post-training pipeline
- Material fidelity
- References
🌏 中文版
This post covers CS336 Spring 2026 Lecture 15: Mid/post-training (SFT/RLHF), taught by Tatsunori Hashimoto on May 18, 2026. Its primary source is the official lecture_15.pdf.
Pretraining produces a base model that continues text. Instruction following requires tighter control. Lecture 15 organizes the mainstream approach into two stages: supervised fine-tuning (SFT) imitates demonstrations, and RLHF optimizes outputs against preferences.
SFT data teaches content and style together
FLAN-like data converts existing NLP tasks into instructions. From Alpaca and OpenAssistant to later tool-use and code datasets, responses become longer and more conversational, adding references, reasoning traces, and tool calls.
These changes do more than add capability. Response length, bullets, tone, refusal format, and citation habits are imitated too, while preference judges often favor longer responses. SFT datasets should separate task correctness, style, and safety rather than relying on one win rate.
The lecture emphasizes a limit: SFT is best at extracting capabilities already present from pretraining, not necessarily injecting unknown tail facts. A small fine-tuning set can teach confident answer form without reliably storing knowledge. For new knowledge, pretraining or mid-training, retrieval, and tools may be better than a few QA pairs.
Mid-training moves instruction-like data into pretraining
Instruction tuning need not occur only as a small final update. Mid-training or two-phase training changes the late-pretraining mixture to include long context, domain data, and instruction-like documents before formal post-training.
The model absorbs the new distribution through next-token prediction and fuller training dynamics. The boundary between mid-training and SFT is not a universal name; it is a combination of data format, token budget, objective, and stage.
Why optimize preferences after imitation
What humans can demonstrate is not always what they prefer when comparing candidates—the generation–verification gap. Preference data typically contains a prompt, chosen response, and rejected response, sourced from experts, crowd workers, AI feedback, or self-training.
Annotator demographics, compensation, guidelines, expertise, and platform workflow all change labels. Pairwise judgments also mix length, style, and cultural or political preference. Data is not an abstract reward source; it is produced by particular judges under particular rules.
PPO learns a reward and updates a policy
Traditional RLHF trains a reward model on pairs, samples responses from a policy, then uses PPO to raise reward while a KL penalty keeps the model near a reference policy. A value model and advantages reduce policy-gradient variance; clipping limits update size.
Its advantage is on-policy exploration of what the current model produces. Its cost is an expensive, unstable system involving rollouts, reward and value inference, and several model states.
DPO writes preference into a direct classification loss
DPO derives from the optimal policy under KL regularization and compares chosen versus rejected log-probabilities relative to a reference. It removes an explicit reward model and on-policy rollout, resembling supervised pairwise optimization operationally.
Simplicity is not universal superiority. DPO learns from a fixed distribution and does not automatically collect failures created by its updated policy. Temperature, reference model, pair quality, and difficulty still matter. PPO versus DPO results are highly contingent on setup.
Reward overoptimization and mode collapse
Optimizing a proxy reward long enough lets a policy exploit reward-model weaknesses: reward rises while actual human preference falls. Entropy can also collapse, producing homogeneous, overly long, templated outputs and losing calibration or diversity from the base model.
Safety SFT or RLHF can cause over-refusal. Evaluation should track helpfulness, harmlessness, factuality, diversity, calibration, and capability retention across checkpoints, not only at the end.
An auditable post-training pipeline
Version SFT prompts and responses, preference pairs, annotation guidelines, reward or judge, and reference model. After SFT, evaluate capability, style, and safety slices. During preference optimization, chart proxy reward and external evaluations against steps and preserve early stopping. High-risk domains need experts and real task outcomes rather than one LLM judge.
Lecture 15's central distinction is that SFT imitates a data distribution while RLHF optimizes a preference proxy. Both are bounded first by data and evaluation.
Material fidelity
This lecture has a Spring 2026 schedule entry and a complete official PDF. This guide covers its SFT data, mid-training, preference data, PPO, DPO, and failure modes.
References
Loading...