All of CS234 assumes the reward is given. The Winter 2026 ethics and society guest lecture (Wanheng Hu, based on material originally developed by Dan Webber) asks, over two sessions, what you really want. The first session splits "alignment" into three targets: the user's intentions, revealed preferences, and objective best interests, with RLHF-driven sycophancy and a personal AI agent as case studies. The second adds a fourth target, what is morally right for people besides the user, and compares three routes: top-down (write principles down), bottom-up (learn from examples), and participatory AI. There's no silver bullet, but alignment can be better or worse.
Part 1 was about an AI setting its own loss and updating its own parameters. Part 2 fills in the other half: AI Agent = Harness + LLM, and the harness can grow too. You can't take a gradient through a harness, so the usual move is to hand it to a language model as a rewriter and keep a pool of candidates, much like a genetic algorithm (OPRO, GEPA, Darwin Gödel Machine; DSPy if you want a ready-made tool). Three extensions follow: updating harness and parameters together beats updating either alone; when the goal changes you have to choose between discarding everything and carrying everything, and editing a harness can cause forgetting too; and the update rule itself can be updated (HyperAgent, Gödel Agent, SEAL), which is meta learning. Hung-yi Lee closes with a new analogy — parameters are genes, context is the neurons — then argues that today's agents lack intrinsic motivation, and that the likeliest source of runaway growth is a gap between the goal humans meant and the goal the AI inferred.
Lecture 13 separates alignment into specification, distribution shift, oversight, and corrigibility, then uses benchmark selection, leakage, and post-hoc selection experiments to show why a final paper cannot audit an autonomous research workflow.