🌏 中文版
Video status: Videos included. Source details
Search assumes successors can be enumerated. Reinforcement learning must act under uncertain outcomes and delayed rewards. Lectures 9–12 establish MDPs and then RL; Project 3 implements value iteration, Q-learning, epsilon-greedy exploration, and approximate Q-learning.
Separate planning from learning
Value iteration knows transition and reward models and computes values through Bellman updates. Q-learning does not require the model in advance; it updates Q-values from (state, action, reward, nextState) experience. Both can produce a policy, but their information sources differ.
Check synchronous updates carefully: every new value in an iteration should come from the previous iteration, not from values modified earlier in the same sweep. For Q-learning, observe learning rate, discount, and exploration separately. A random epsilon-greedy action gathers information; it is not a malfunction.
From tables to features
Tabular Q-learning stores every state-action pair separately and generalizes poorly in a large Pacman state space. Approximate Q-learning expresses Q-values through features and weights so related situations share experience. First verify that an identity extractor behaves like the tabular version, then introduce meaningful features. This isolates the effect of representation while preserving the update rule.
Hand-calculate one Bellman update in a tiny Gridworld before running a single autograder case. Finally compare policies before and after exploration is disabled. Do not stop at average score; explain why the agent selects its action.
Series navigation: Previous: CSPs and multi-agent search | Next: Bayes nets and Ghostbusters
Course video sources
The official Spring 2026 schedule and the official YouTube playlist ([CS188 SP26] Live Lectures, 28 videos) were checked live on 2026-10-10; the lecture recording embedded here is listed there as a “Recording”. No unverified timestamp is supplied.
Original videos: CS188 Spring 2026 Lecture 9: MDPs I
Course and recording entries:
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): sampled check (only the first ~5,000 characters of the transcript were read): the embedded video is Lecture 9, on MDPs I; the transcript introduces MDPs by recalling expectimax chance nodes and contrasts MDP assumptions with search, matching this article. The remaining sections of this article cover other lectures in the unit and the project, not this video.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. The official Spring 2026 schedule and YouTube playlist were checked live and list the embedded lecture recording, so the status is now Videos included.
- 2026-10-10: Checked the video content against its transcript. Confirmed the embedded video is Lecture 9 (MDPs I) and fits the article scope; no video claims needed correcting.
References
Loading...