Week 6 processes natural language: N-gram conditional probability & smoothing, CFG syntax parsing with CYK, TF-IDF vector retrieval, attention mechanism & Transformer basics. Projects: Parser (syntactic generation) and Questions (TF-IDF QA system).
Lecture 17 first decides how text becomes tokens, then uses N-grams to turn sequence probability into conditional probabilities estimated from corpus counts. Tokenization is the first design decision about what a model can see.
Lecture 18 truncates the chain rule with an N-gram Markov assumption, estimates probabilities from corpus counts, and contrasts greedy, categorical, and temperature sampling. The real bottlenecks are zero probability for unseen contexts and a fixed window.
Lecture 19 builds a minimal next-token model from two embedding matrices, dot-product similarity, softmax, and cross-entropy. Shared vector parameters replace the isolated count cells of an N-gram table.
Week 2 builds three layers: a token vocabulary with BPE, sequence comparison with dynamic-programming edit distance, and probability approximation with n-grams; PA1 turns regex and BPE into executable work.
Lecture 7 decomposes pretraining into scalable data, subword tokenization, three model objectives, and in-context learning. A general self-supervised objective yields reusable representations; downstream signals specify their use.
Lecture 4 defines a language model as a next-word probability distribution, then uses an RNN to compress an arbitrarily long prefix. It also exposes recurrence's central cost: information and gradients travel one time step at a time.
The final lecture is not a complete LLM-training tutorial. It studies data efficiency under fixed data and abundant compute, revisiting epochs, batches, ensembles, self-training, and conditions for synthetic continued pretraining.
The first four units make text countable, representable, and classifiable; A1 then moves from n-grams and perceptrons to an NBOW MLP.