Skip to content
All tags

#natural-language-processing

3 posts

CMU 07-280 Lecture 17: From Tokenization to N-gram Language Models

Lecture 17 first decides how text becomes tokens, then uses N-grams to turn sequence probability into conditional probabilities estimated from corpus counts. Tokenization is the first design decision about what a model can see.

CMU 07-280 Lecture 18: How N-grams Train, Sample, and Fail

Lecture 18 truncates the chain rule with an N-gram Markov assumption, estimates probabilities from corpus counts, and contrasts greedy, categorical, and temperature sampling. The real bottlenecks are zero probability for unseen contexts and a fixed window.

CMU 07-280 Lecture 19: Turning Next-token Prediction into Geometry

Lecture 19 builds a minimal next-token model from two embedding matrices, dot-product similarity, softmax, and cross-entropy. Shared vector parameters replace the isolated count cells of an N-gram table.