This lecture takes a large language model apart in three layers: tokenization turns text into token IDs, the chain rule splits the probability of a whole text into a sequence of next-token conditionals, and training sums the cross-entropy at every position. The second half opens up the Transformer: inner products of queries and keys go through a softmax to give weights, the weights average the values, and a causal mask blocks future positions. The cost of a single attention layer grows with the square of the sequence length T.
During generation, every head has to keep the keys and values of all past positions in GPU memory, and this KV cache caps the batch size. GQA lets τ query heads share one set of keys and values, shrinking the cache to 1/τ; sliding window attention cuts the T² compute down to T·w; MoE runs only 8 of 128 experts for each token. The second half covers how to use a trained model: few-shot and zero-shot prompting leave the parameters alone, SFT changes them, and its loss is computed only on the answer tokens.
Cross-attention reads another representation, DiT is a diffusion-backbone choice, and Q-Former uses learned queries to extract or bridge information. They can coexist rather than being three competing products. The Lecture 14 lesson handles Prompt-to-Prompt and classifier-free guidance in detail.
Long-context evaluation asks whether a model can process, retrieve, and use distant information. An SSM specifies sequence processing through state updates. SSM efficiency does not establish reliable document retrieval, and a large attention window does not guarantee relevant evidence is used.
Standard attention writes the N×N score matrix out to HBM and reads it back, and most of its time goes to that traffic. FlashAttention uses tiling plus softmax rescaling so each block finishes inside SRAM, and the backward pass recomputes instead of storing. Tri Dao's guest slides for 11-868 give one set of numbers: the backward pass does 13% more FLOPs, 9x less HBM traffic, and runs 6x faster. FA3 and FA4 follow the same theme: when the hardware changes, the bottleneck moves, and the algorithm has to move with it.
Across two lectures and more than 200 slides, Google's Srinath Mandalapu traces one attention computation from Python down to TPU VLIW instructions. L12 covers the JAX ecosystem, the memory and compute units of TPU Ironwood, and how XLA compiles attention into three fused kernels. L13 covers what XLA cannot do: using Pallas to control movement between HBM and VMEM yourself, writing FlashAttention, then adding block sparsity to get Splash Attention. The ideas match the GPU version. The difference is that on TPU the compiler does most of the scheduling, and Pallas is how you take loops and block sizes back into your own hands.
Lecture 8 of CS231N Spring 2026 starts from the bottleneck in RNN translation models, abstracts attention into an operation on sets of vectors, builds up to self-attention, masking, and multiple heads, and shows the whole layer is four matrix multiplies. A Transformer block is self-attention, LayerNorm, residual connections, and an MLP; ViT turns a 224×224 image into 16×16 patches used as tokens. The lecture closes with four common post-2017 changes: Pre-Norm, QK-Norm, SwiGLU, and MoE.
Lecture 15 has four parts. Extending context: interpolating RoPE stretches LLaMA from 2k to 32k, and LongLoRA's shifted sparse attention makes long-context fine-tuning cheap. Evaluation: lost-in-the-middle, Needle-in-a-Haystack, and LongBench. Efficient attention: the KV cache grows linearly with length. StreamingLLM finds that the first few tokens act as attention sinks, and keeping them plus a recent window gives stable generation. DuoAttention keeps a full KV cache only for a few retrieval heads. Quest keeps the whole KV cache but reads only the most critical pages for each query. The last part moves beyond Transformers: Mamba replaces attention with a selective SSM, and Jamba mixes the two.
L05 reads the whole Transformer with two linear-algebra rules: matrix multiplication is row-times-column dot products, and a row vector times a matrix is a linear combination of the matrix's rows. With those, attention is 'dot the query with every key, softmax into weights, take a weighted average of the values,' or softmax(QKᵀ/√d_k)V in batch form. Dividing by √d_k just pulls the numbers back toward 0 so softmax doesn't become winner-take-all. Then come multi-head attention, encoder versus decoder, masking, positional encoding as a set of sin/cos clocks, and ResNet-style residuals with layer normalization. No homework this week.
"Look over there" is 3 tokens, the Chinese version 4, the Japanese version 12. When output length does not track input length, the classifier trick of adding an FFN at the end fails. This 33-slide deck starts from encoder-decoder models, works through vanishing gradients in RNNs and the three LSTM gates, and ends with attention fixing both long-range memory and parallelism, including why scores are divided by √d.
An RNN translator has to squeeze the whole source sentence into one vector, and long sentences don't fit. Attention lets the decoder look back at every input position each time it produces a word, score each one, and take a weighted sum. Call the scorer the query, the thing being scored the key, and the thing being averaged the value, and you have dot-product attention. The Transformer goes one step further: the input attends to itself, recurrence disappears, and in return you get parallelism and a constant path length. The price is that position information has to be added back by hand.
In week 3 of ML 2026, Hung-yi Lee spends the first half of the inference lecture on one technique: Flash Attention. A GPU's execution units are fast, but their workbench (on-chip SRAM) is tiny, so data has to be carried to and from the warehouse (HBM). The carrying is the bottleneck. A naive softmax makes several round trips to the warehouse. Flash Attention assumes the current maximum is Amax, then multiplies by a correction factor when a larger value shows up. That lets it find the maximum, build the denominator, and compute the weighted sum in one pass, without ever materializing the attention weights. The output is identical to standard attention, no retraining is needed, and the cost is a little extra compute and a little brain strain.
KV Cache stores the keys and values already computed so decode does not recompute them, but every token costs memory. For Gemma 2 27B that is about 0.72MB per token, so an 80GB A100 holds only about 114k tokens. Hung-yi Lee then walks through ways to shrink it: let queries share keys and values (MQA, GQA), compress keys and values into one vector without ever decompressing (MLA), limit the attention span (Sliding Window, StreamingLLM), and drop keys and values nobody attends to (Scissorhands, H2O). He ends with cross-conversation prompt caching: it only hits when the prefix is identical, so a system prompt should put stable content first.
Self-attention on its own cannot tell "you hit me" from "I hit you", so the model needs position information from somewhere else. Hung-yi Lee's lecture goes from sinusoidal absolute positions to ALiBi and T5's relative biases, then to RoPE, which Llama, Qwen and Gemma all use. The second half covers train-short-test-long: RoPE breaks when it rotates to angles it never saw in training, which led to Position Interpolation, NTK-Aware scaling, YaRN, Dynamic Scaling and LongRoPE. The final twist is NoPE: causal attention in a decoder-only model already carries position information, and you can even drop the positional embedding after training.
CME295 Lecture 1 threads a single sentence, "A cute teddy bear is reading.", through the whole class: split it into tokens, turn them into vectors, see why an RNN can't hold on to long sentences, then translate it into French with self-attention and an encoder-decoder. The 2026 edition drops the entire section on NLP tasks and evaluation metrics and opens instead with a timeline running from 2017 to the agent era.
CME295 Lecture 2 takes the original Transformer apart and refits it: position information moves from "added to the embedding" to RoPE's "rotate Q and K inside attention"; attention gets cheaper with sliding windows and MQA/GQA; models split into encoder-only, encoder-decoder, and decoder-only families; and the second half dissects BERT's MLM (15% of tokens) and NSP pretraining. The 2026 edition folds all of this into a single "Large Language Models" lecture, and BERT is no longer a syllabus item.
The first three parts of CS224U's Spring 2023 contextual representations unit start with examples like "break" and "crane" to show why static word vectors were never going to be enough. They then build a Transformer block step by step on the three words "The Rock rules." Only attention connects the columns; every other step runs on each column independently. Finally, two questions sort three positional encoding schemes: Do you have to fix the set of positions ahead of time? Does the scheme get in the way of generalizing to new positions? Absolute encoding fails both, sinusoidal encoding passes the first, and the relative encoding of Shaw et al. (2018) passes both.
Lec 21 starts with what CNNs lack: only the top layers see the whole image. It then builds up through TF-IDF, RNN image captioning, and soft attention. Lec 22 derives self-attention from a "soft dictionary lookup": three linear layers produce Q, K, and V, the output is SoftMax(QKᵀ/√D)V, and multiple heads, an MLP, residual connections, and LayerNorm turn it into a transformer layer. Attention itself ignores order, so you need positional encodings. Discussion 10 has you compute QKV by hand and prove why the scores are divided by √D.
HW4 Problem 1 (40 pts) takes self-attention apart in five steps: a 2×2 hand calculation, why we divide by √dk, permutation equivariance without positional encoding, single-head attention in pure NumPy, and multi-head attention in PyTorch with an attention heatmap on synthetic data.
Week 6 is about getting computers to handle natural language: parse sentences with hand-written context-free grammar rules, then switch to statistics, from n-grams and Markov-chain text generation to a Naive Bayes classifier for review sentiment; turn words into vectors with word2vec, process whole sequences with an RNN encoder–decoder, and finish with attention and the Transformer. Includes section-by-section time links, plus an approach, common mistakes and ways to check your work for the Parser and Attention projects.
A Transformer is not an architecture from nowhere: tokens discretize data, attention does soft aggregation, positional codes restore order. Seen next to MLPs/CNNs/GNNs, all of them are special cases of 'weighted aggregation over neighbors'.
The core of the Transformer is self-attention: for each token, the model computes how relevant every other token is, then takes a weighted sum. This lets the model reach across distance to figure out that 'it' refers to 'cat' not 'mat' — and is the foundation for how it handles long documents.
Lecture 4 studies two kinds of sparsity: linear/recurrent attention reduces sequence-length cost, while MoE activates only part of a model for each token. Both turn saved FLOPs into routing, balancing, communication, and kernel problems.
Lectures 19–25 connect rational decisions and VPI to machine learning, while Project 5 uses PyTorch for regression, classification, CNNs, attention, and an optional character-GPT.