Inference Methods: After Training, a Model Can Still Get Smarter by Computing More | MIT 6.7960 Lecture 24
A trained model is not limited to one forward pass. Greedy sampling does not guarantee the sequence the model itself considers most likely (in this post's example it gets 0.30 where the best is 0.36), so sampling several times and keeping the best already helps; swap the scoring criterion for a verifier and you can pick answers that are actually correct. Feed the good answers found by search back into training and you have the loop behind AlphaGo and STaR, which Isola speculates is also how o1 works.