CS224R Spring 2026 HW2 has three parts. First, tabular Q-learning on a 5×4 gridworld shows how reward design changes the learned path. Second, GAE plus PPO clipping tackles a hammer task that pays 1 only on completion. Third, an off-policy actor-critic with BC pretraining, a critic ensemble, and a higher UTD ratio, followed by a comparison of the two learning curves. The handout, starter code, and compute guide are public, but the assignment supports only Modal, and course credits go only to enrolled students.
Q-learning drops the actor from actor-critic: learn the optimal Q-function directly and act by taking the argmax. The price is that convergence is not guaranteed; even linear Q can diverge. Lecture 6 of CS224R pulls it back with three engineering tricks: a target network that holds the targets still, Double Q that separates choosing an action from valuing it to curb overestimation, and n-step returns that trade a little bias for speed.
Once you can evaluate a policy, the next step is to improve it while you collect data. CS234 Lecture 4 goes like this: ε-greedy keeps policy improvement monotonic; GLIE says how much to explore and when to stop; Q-learning converges to Q* under GLIE plus Robbins–Monro step sizes; and finally the table becomes a parameterized Q̂(s,a;w) trained by SGD on MC, SARSA, or Q-learning targets. The price is the deadly triad: function approximation, bootstrapping, and off-policy learning together can oscillate or diverge.
Every model in the first 12 lectures learned from training data that people prepared. L13 asks a different question: when there is no right answer, only a signal of how well you did, how does a computer learn? Tsai starts from AlphaGo and Breakout and splits the field in two. Value-based methods learn a Q function that scores each action (Deep Q-Learning, TD, experience replay, ε-greedy); policy-based methods learn the action directly (policy gradient, actor-critic). The second half returns to LLMs: ChatGPT trains a reward model from human rankings and then runs RLHF with PPO, while DeepSeek has the computer check math answers automatically and uses that as the reward. Week 13 homework is the final project proposal.
HW6 Problem 3 (20 pts) has you write a tabular Q-learning agent for Swingy Monkey, a Flappy Bird-like game. The minimum bar is scoring over 50 at least once within 100 epochs, plus one improvement of your choice. Problem 5 (10 pts) is a 250-word ethics question: assuming a social platform's users grow more politically extreme, use RL concepts to explain how the choice of reward function might have contributed.
Week 4 covers three forms of machine learning. Supervised learning fits a hypothesis function to labeled data: classify by the k nearest neighbors, or draw a decision boundary with a set of weights (perceptron learning rule, logistic regression, support vector machines), then use loss functions, regularization and cross-validation against overfitting. Reinforcement learning uses Q-learning to update the estimated value of each state and action from rewards, with ε-greedy keeping some probability of random exploration. Unsupervised learning clusters with k-means. With time links for every section, plus the approach and ways to check your work for the Shopping and Nim projects.
Lecture 22 keeps the MDP structure but removes known transitions and rewards. TD learning updates value from one sample, and Q-learning uses an off-policy target to learn optimal action values directly.
Lectures 9–12 and Project 3 use the same Gridworld to contrast value iteration with a known model, Q-learning from unknown dynamics, and approximate Q-learning that generalizes through features.
L5–10 build the deep-RL core through policy- and value-based routes; HW2 is CPU-friendly, while HW3's Atari and HalfCheetah runs can require hours of GPU time.
The third reason Go can't be learned with supervision is the interesting one: the ground truth itself is ill-defined — the strongest human doesn't play their best moves every day, and even their best move isn't optimal. The last 15 minutes or so map RLHF fully back onto RL: the agent is the model being fine-tuned, the action is the next token, an episode is one full generation, and the reward is extremely sparse.