🌏 中文版
Video status: Videos included. Source details
The official schedule assigns L11–18 to Variational Inference, VI in RL, Control as Inference, LLM RL, two lectures on Model-Based RL, and two on Offline RL. Together they ask what signal an agent can trust when data, models, and objectives are incomplete.
L11–14: control as inference
L11–13 establish latent-variable and variational-inference machinery, then express optimality as a probabilistic event. Reward, trajectory distributions, and entropy enter one language. Section 6 supports the derivation; Section 7 connects IRL and LLM RL.
L14 applies policy optimization to token policies and verifiable rewards. HW4 implements REINFORCE and GRPO on format-copy and harder math tasks.
It is not a laptop-first exercise. The homework compute ledger owns the supporting H100 and required-run details.
L15–16: learning or using dynamics
Model-based RL learns or uses dynamics, then plans or improves a policy through that model. Data reuse is the advantage; compounding model error is the danger. For Section 8, draw three loops—data collection, model learning, and planning or policy learning—and mark every possible distribution shift.
L17–18: learning from a fixed dataset
Offline RL cannot collect corrective interactions. Out-of-distribution actions can therefore receive overestimated values. HW5 implements SAC+BC, IQL, and FQL on OGBench tasks. Its long-running experiments and tuning burden are detailed in the homework compute ledger.
A reduced self-study path
Read the slides and sections first. For HW4, start with the smallest format-copy run. For HW5, choose one task, seed, and baseline, and verify loading, evaluation, and checkpoints before scaling. The goal is understanding failure modes, not reproducing course-only compute support.
See the series overview's access boundary for current course assets and the proper use of historical video.
Course video sources
Checked live on 2026-10-10: the instructor published the Spring 2026 lecture recordings on the RAIL YouTube channel (playlist “CS 185/285: Deep Reinforcement Learning (Spring 2026)”, 27 videos, public since 2026-08-15). Lecture numbers match the official slide list, so these recordings correspond to the lectures this post covers. The course syllabus still says recordings are on bCourses and the course site still links the Fall 2023 playlist; neither is needed to watch these. Two of the lectures are embedded here; the rest are in the playlist.
Original videos: CS 185/285 (Spring 2026): Lecture 11, Variational Inference、CS 185/285 (Spring 2026): Lecture 17, Offline Reinforcement Learning
Course and recording entries:
- Official course and recording entry
- CS 185/285: Deep Reinforcement Learning (Spring 2026) — official RAIL YouTube playlist (27 videos)
Checked: 2026-10-10.
Content check: verified against the video transcript (2026-10-10): the Lecture 11 transcript covers latent-variable models, expected log likelihood, deriving the ELBO with an approximate posterior (Jensen's inequality), the relation between entropy and KL divergence, EM as a special case, and uses of latent variables in flow-matching policies, model-based RL and RLHF; the Lecture 17 transcript covers the core offline-RL difficulty of distributional shift, policy constraints (including a discussion of KL direction and actor-critic plus BC) and pessimism. This matches the note's inference view and offline-RL out-of-distribution framing; the specific HW4/HW5 algorithms (GRPO, IQL, FQL, etc.) are not discussed in the transcripts.
Update Log
- 2026-10-10: Added explicit video status and checked recording sources and access notes.
- 2026-10-10: Rechecked video status. Found the public Spring 2026 CS 185/285 YouTube playlist on the RAIL channel; embedded two matching lectures and changed the status from official entry only to Videos included.
- 2026-10-10: Checked the video content against its transcript. Confirmed the two videos are Lectures 11 and 17 and match the note; the HW algorithms are not in the videos.
References
Loading...