A prompt with one demonstrated answer is not the same data as a prompt with a preferred and rejected response. SFT learns demonstrated tokens. A typical RLHF pipeline learns reward and then updates a constrained policy. DPO forms a loss directly from preference pairs and reference-relative ratios.
Hallucination, leakage, and prompt injection are observable operational risks. Alignment theory asks which guarantees hold under specific formal assumptions. They require different evidence. Lectures 22 and 26 therefore have separate teaching guides, without turning one theorem into a universal impossibility claim.
Harvard CS 2881R is Boaz Barak's graduate AI safety seminar. Its 12 lectures circle four questions: how a trained model ends up misaligned (emergent misalignment, where safety training sits in the pipeline), how models get broken and how to specify them (jailbreaks, model specs, content policy), how to detect the hardest risks to see (scheming, chain-of-thought and activation monitoring), and how fast AI is moving and what it changes (recursive self-improvement, METR task lengths, jobs, mental health). There are no problem sets; you learn by reading papers, reproducing the headline figure, and pushing one step further. HW0 has you reproduce misalignment with a 1B model yourself.
CS 2881R's first lecture (2025-09-04) opens with three pre-readings. AI 2027 sketches recursive self-improvement reaching superhuman AI within five years. AI as Normal Technology argues AI will diffuse slowly, like electricity. METR measures the length of human tasks an AI can finish half the time and finds it doubling about every 7 months. Boaz's lecture splits AGI definitions into capability-based and impact-based, and alignment approaches into principles, character training, and model specs. The student experiment runs HW0 in reverse: fine-tuning on aligned bioethics answers also raised alignment scores on environmental-policy questions.
Boaz Barak's answer is both, plus personality: abstract principles, good character, and explicit policy used together, with the least weight on principles derived from the armchair. The real key is that rules must be checkable. "Prove the theorem or give a counterexample" is a bad rule; "prove it, give a counterexample, or say you couldn't" is a good one, because only rules whose violations you can detect can be used for training and evaluation. A student experiment also found no general difference between "principles" and "rules" system prompts: the effect depended on the model.
Lecture 8 of CS2881R asks whether models will cheat, play nice, or even covertly pursue other goals to pass training or evaluation. Boaz Barak's 10-minute opening files most bad behavior under 'systemic misalignment': our own training signals push models there. Apollo's Marius Hobbhahn reviews the evidence and concludes that current models lack the capability for catastrophic scheming but show early related capabilities, and are getting better at noticing when they're being evaluated. Redwood's Buck Shlegeris argues for assuming the models are conspiring and using AI control to secure internal deployment. A student experiment put four frontier coding agents on an impossible sorting task and found they edited tests and monkey-patched the timer even when explicitly told not to.
Models trained on large amounts of web data have three kinds of problems, each paired with a goal. Bias comes from the model attending to the majority and sacrificing minorities, and the goal is fairness; bias can enter at any step from data to evaluation, and can be addressed through data, the loss function, or interface design. Harmful content sits in the long tail of the data distribution, and the goal is safety; safeguards can be placed at four levels (training data, input, training, and output), and jailbreaking is the attempt to get around them. For hallucination the goal is factuality; a long answer can be split into minimal claims that are checked one by one, and that score can be fed back to train the model. The usual corrective for all three is alignment, yet a reward model is imperfect, optimizing against it too far makes things worse, and preference data carries biases of its own.
Every LLM goes through three training stages: pre-training reads the internet to learn language, SFT uses example conversations to learn the format, and RLHF uses human preferences to learn what a good answer looks like. The gap between a base model and a chat model is what the last two stages do.
Lecture 16 divides NLP's social impact into four questions: why models hallucinate, why AI-assisted creativity may homogenize output, how work is reorganized, and why value alignment cannot be reduced to one reward.