Table of Contents
🌏 中文版
The through-line of units 05–07 is information flow across positions. A sequence model updates state; seq2seq separates encoder and decoder; attention lets the decoder select source positions directly; a Transformer replaces the recurrent path with parallel attention blocks.
Compare information paths, not labels
An RNN compresses history into a fixed-size state, forcing distant information through repeated updates. Seq2seq structures input and output but can retain a context bottleneck. Attention creates content-addressed shortcuts. Transformers must then account explicitly for position, masking, normalization, and residuals.
While reading equations, label every tensor shape and draw token-to-token dependencies. This catches causal-mask, head-reshape, and residual mistakes faster than memorizing the words query, key, and value.
A2 turns architecture into testable components
Assignment 2 implements BPE, RMSNorm, SiLU, SwiGLU, RoPE, scaled dot-product attention, causal multi-head attention, Transformer blocks, and an LM, followed by training utilities, FLOPs, and memory estimates. Its starter repository is public.
Preserve the component tests. Verify shapes, masks, and numerical stability on tiny tensors before training TinyStories. Without a GPU, reduce layers, hidden size, and context length; correctness remains testable. Hidden tests and official Gradescope remain unavailable, so local success is not official completion.
References
Loading...