Skip to content
All tags

#temporal-difference

2 posts
aiguideReading Stanford CS234 Reinforcement Learning

Evaluating a Policy Without a Model — MC, TD(0), and Certainty Equivalence | Stanford CS234 Topic Guide

If you don't know the transition probabilities or rewards, how do you estimate what a policy is worth? CS234 Lecture 3 gives three answers. Monte Carlo averages full-trajectory returns: unbiased, high variance, and it has to wait for the episode to end. TD(0) targets one real reward plus the next state's estimate: biased, lower variance, and it updates every step. Certainty equivalence estimates a model and then runs dynamic programming: the most data-efficient and the most expensive to compute. The AB example at the start of Lecture 4 makes the difference plain: on the same data, MC says V(A)=0 and TD says V(A)=0.75.

#cs234#ai-course#stanford#reinforcement-learning#temporal-difference#mdp
aideep-diveReading CMU 07-280 AI & ML I

Q-learning When Dynamics Are Unknown | CMU 07-280 Lecture 22

Lecture 22 keeps the MDP structure but removes known transitions and rewards. TD learning updates value from one sample, and Q-learning uses an off-policy target to learn optimal action values directly.

#cmu#ai-course#reinforcement-learning#q-learning#temporal-difference