CMU 07-280 Lecture 21: How Bellman Equations Solve Markov Decision Processes
Lecture 21 formulates stochastic sequential decisions as an MDP with known dynamics, defines value and Q-values through Bellman backups, and solves for an optimal policy with value or policy iteration.