🌏 中文版
Video status: Videos included. Source details
Lectures 5–10 form the algorithmic core. The official agenda covers Policy Gradients, Actor Critic, Value-Based RL, Q-learning in Practice, and two Advanced Policy Gradients lectures. Read them by asking what is estimated, where data comes from, and how bias trades against variance.
Policy-based methods
L5 derives policy gradients from a trajectory objective. Reward-to-go, baselines, and advantages reduce variance without changing the desired objective. L6 introduces actor-critic: a critic supplies the actor's update signal, potentially adding function-approximation bias. Sections 3 and 5 connect and extend these ideas.
HW2 experiments with reward-to-go and neural-network baselines. It is a CPU-first assignment; the homework compute ledger owns the timing and hardware details. For self-study, hold the environment fixed, run three seeds, and retain both individual curves and their mean.
Value-based methods
L7–8 move from Bellman backups to DQN and its stability machinery. L9–10 return to advanced policy-gradient methods. Section 4 places DQN beside SAC; compare their update targets, replay buffers, target networks, and entropy terms.
HW3 implements DQN and SAC. Its starter code spans cheap and expensive environments; see the homework compute ledger for GPU estimates. Validate losses, replay, and evaluation in a small environment first.
Completion check
Explain why policy gradients have high variance, how a critic trades variance for bias, why DQN uses replay and target networks, and what entropy contributes to SAC. If any answer is vague, return to the derivation and smallest experiment before spending more compute.
Course video sources
Checked live on 2026-10-10: the instructor published the Spring 2026 lecture recordings on the RAIL YouTube channel (playlist “CS 185/285: Deep Reinforcement Learning (Spring 2026)”, 27 videos, public since 2026-08-15). Lecture numbers match the official slide list, so these recordings correspond to the lectures this post covers. The course syllabus still says recordings are on bCourses and the course site still links the Fall 2023 playlist; neither is needed to watch these. Two of the lectures are embedded here; the rest are in the playlist.
Original videos: CS 185/285 (Spring 2026): Lecture 5, Policy Gradients、CS 185/285 (Spring 2026): Lecture 7, Value-Based RL
Course and recording entries:
- Official course and recording entry
- CS 185/285: Deep Reinforcement Learning (Spring 2026) — official RAIL YouTube playlist (27 videos)
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): the Lecture 5 transcript derives the policy gradient from the trajectory objective and covers reward-to-go and baselines (variance reduction that stays unbiased), and mentions policy gradients training language models; the Lecture 7 transcript explains that removing the actor and keeping only the critic gives Q-learning, then moves from policy iteration, value iteration and fitted value iteration to fitted Q iteration, with replay buffers, epsilon-greedy and Boltzmann exploration, and previews stabilizing tricks for the next lecture. This matches the note; the word "advantage" does not appear in the transcript, DQN tricks such as target networks belong to the next lecture, and the HW2/HW3 details are the site's own synthesis.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Found the public Spring 2026 CS 185/285 YouTube playlist on the RAIL channel; embedded two matching lectures and changed the status from official entry only to Videos included.
- 2026-10-10: Checked the video content against its transcript. Confirmed the two videos are Lectures 5 and 7 and match the note; noted that DQN stabilizing tricks are in the next lecture.
References
Loading...