How the model reads a history, learns from targets, and chooses its next token are different questions. Attention mixes information; the training objective updates parameters; decoding selects output under a fixed model. Keeping them distinct prevents confusing a sampling change with a model update.
A guide to the Transformer unit of Prof. Hung-Yu Kao's NLP course at NTHU (Fall 2025). The slides start from two RNN problems: words interact only across O(N) steps, and time steps cannot run in parallel. Self-attention lets every word look at every other word and computes it all in one matrix multiplication, fixing both at once. The cost is that the model can no longer tell word order, so sinusoidal positional encoding is added. The lecture then assembles multi-head attention, Add & Norm, feed forward, cross-attention, masked attention and teacher forcing, and closes with GPT-2, ViT and four variants to show how far the architecture went.
Lec 21 starts with what CNNs lack: only the top layers see the whole image. It then builds up through TF-IDF, RNN image captioning, and soft attention. Lec 22 derives self-attention from a "soft dictionary lookup": three linear layers produce Q, K, and V, the output is SoftMax(QKᵀ/√D)V, and multiple heads, an MLP, residual connections, and LayerNorm turn it into a transformer layer. Attention itself ignores order, so you need positional encodings. Discussion 10 has you compute QKV by hand and prove why the scores are divided by √D.
HW4 Problem 1 (40 pts) takes self-attention apart in five steps: a 2×2 hand calculation, why we divide by √dk, permutation equivariance without positional encoding, single-head attention in pure NumPy, and multi-head attention in PyTorch with an attention heatmap on synthetic data.
The core of the Transformer is self-attention: for each token, the model computes how relevant every other token is, then takes a weighted sum. This lets the model reach across distance to figure out that 'it' refers to 'cat' not 'mat' — and is the foundation for how it handles long documents.
Lecture 5 moves from the long-range and sequential bottlenecks of RNNs to self-attention and the Transformer. It shortens information paths and enables parallel computation, at the price of quadratic attention and separately encoded position.