The first 55 minutes of this lecture write sequential decision making as an MDP (states, actions, transition probabilities, rewards, discount factor) and show that the Bellman equation for a fixed policy is a system of linear equations. The last 20 minutes derive policy gradient: ∇p = p∇log p turns the gradient back into an expectation, the transition probabilities drop out of the gradient because they don't contain θ, and the lecture stops at the formula "sum of log-probability gradients × sum of discounted rewards", leaving variance reduction for the next lecture.
The first lecture of CS224R Spring 2026 does three things: covers logistics, explains why deep RL is worth learning, and turns 'behavior' into something you can learn. The core is a set of definitions (state, action, trajectory, reward, policy) and one objective: maximize expected total reward. It ends on an example: fit ℓ2 regression to drivers where some change lanes and some go straight, and the policy learns their average, a half lane change nobody demonstrated. That problem is where L2 starts.
CS234's Winter 2026 Assignment 1 is worth 68 points across four questions: an inventory MDP where the horizon and discount change the optimal policy (8), a traffic example where a proxy reward makes the AI car refuse to merge (5), bounding a greedy policy's performance with the Bellman residual (30), and hand-written value iteration and policy iteration on RiverSwim (25). The three written questions all drill one idea: the reward, γ, and value function you write down may not be the goal you think they are.
Lecture 1 of CS234 Winter 2026 first answers what RL is: learning from experience to make good decisions under uncertainty. It usually involves four things at once: optimization, delayed consequences, exploration, and generalization. A seven-cell Mars rover world then builds from a Markov process to a Markov reward process, defining return, the value function, and the discount factor, and ends with the Bellman equation for an MRP. You can solve it with a matrix inverse or iterate with dynamic programming. Add actions and you get an MDP, where the next lecture starts.
Lecture 2 of CS234 Winter 2026 assumes the world model is known and asks how to compute the best policy. An MDP plus a policy is an MRP, so a policy can be evaluated by iterating a Bellman backup. Policy iteration alternates evaluation and improvement, and the slides prove each round is no worse than the last, so it stops within |A|^|S| rounds. Value iteration takes another route: apply the Bellman optimality operator over and over. For γ < 1 that operator is a contraction, so value iteration always converges. The lecture ends with finite horizons, where the best policy usually depends on how many steps remain.
If you don't know the transition probabilities or rewards, how do you estimate what a policy is worth? CS234 Lecture 3 gives three answers. Monte Carlo averages full-trajectory returns: unbiased, high variance, and it has to wait for the episode to end. TD(0) targets one real reward plus the next state's estimate: biased, lower variance, and it updates every step. Certainty equivalence estimates a model and then runs dynamic programming: the most data-efficient and the most expensive to compute. The AB example at the start of Lecture 4 makes the difference plain: on the same data, MC says V(A)=0 and TD says V(A)=0.75.
HW6 Problem 2 (15 pts) hands you a 4×5 Gridworld where moves can slip and rewards arrive only when you leave a cell. You write one step each of policy evaluation, policy iteration, and value iteration in the notebook, watch how the discount factor γ reshapes the policy, and finally ask whether this is a sensible model of the robot task at all. The rules of the world are fully known, so this is planning, not learning yet.
Lectures 9–12 and Project 3 use the same Gridworld to contrast value iteration with a known model, Q-learning from unknown dynamics, and approximate Q-learning that generalizes through features.