CS336 Lecture 8: Align ZeRO, FSDP, and 3D Parallelism with Hardware Topology
Lecture 8 moves from parallel primitives to system design: ZeRO progressively shards optimizer state, gradients, and parameters; TP, PP, SP, and EP split width, depth, sequence, and experts. Their composition must follow topology and dynamic activation memory.