Control Without a Model — ε-greedy, GLIE, Q-learning, and Function Approximation | Stanford CS234 Topic Guide
Once you can evaluate a policy, the next step is to improve it while you collect data. CS234 Lecture 4 goes like this: ε-greedy keeps policy improvement monotonic; GLIE says how much to explore and when to stop; Q-learning converges to Q* under GLIE plus Robbins–Monro step sizes; and finally the table becomes a parameterized Q̂(s,a;w) trained by SGD on MC, SARSA, or Q-learning targets. The price is the deadly triad: function approximation, bootstrapping, and off-policy learning together can oscillate or diverge.