🌏 中文版
The through-line of units 05–07 is information flow across positions. A sequence model updates state; seq2seq separates encoder and decoder; attention lets the decoder select source positions directly; a Transformer replaces the recurrent path with parallel attention blocks.
Compare information paths, not labels
An RNN compresses history into a fixed-size state, forcing distant information through repeated updates. Seq2seq structures input and output but can retain a context bottleneck. Attention creates content-addressed shortcuts. Transformers must then account explicitly for position, masking, normalization, and residuals.
While reading equations, label every tensor shape and draw token-to-token dependencies. This catches causal-mask, head-reshape, and residual mistakes faster than memorizing the words query, key, and value.
A2 turns architecture into testable components
Assignment 2 implements BPE, RMSNorm, SiLU, SwiGLU, RoPE, scaled dot-product attention, causal multi-head attention, Transformer blocks, and an LM, followed by training utilities, FLOPs, and memory estimates. Its starter repository is public.
Preserve the component tests. Verify shapes, masks, and numerical stability on tiny tensors before training TinyStories. Without a GPU, reduce layers, hidden size, and context length; correctness remains testable. Hidden tests and official Gradescope remain unavailable, so local success is not official completion.
A mask determines which positions are visible
This is an invented teaching example, not an official assignment solution.
For invented tokens A, B, C at positions 1, 2, 3, causal attention permits position 1 to read A, position 2 to read A and B, and position 3 to read all three. It includes the current position but excludes future positions. If position 2 reads C, training has used information unavailable at that inference step.
Check and answer: How many entries of the three-by-three mask are visible? 1+2+3=6. This example has no padding; padding requires its own masking considerations.
Changelog
- 2026-10-11: Added an invented example, worked calculation or reasoning, and a comprehension check with an answer.
References
Loading...