Skip to content
All tags

#distributed-training

2 posts
ai deep-dive Reading Stanford CS336

CS336 Lecture 7: Build Data, Tensor, and Pipeline Parallelism from Collectives

Lecture 7 starts below FSDP APIs, building a communication language from broadcast, all-reduce, all-gather, reduce-scatter, and all-to-all before assembling data, tensor, and pipeline parallelism.

#cs336#distributed-training#pytorch#gpu#parallelism
ai deep-dive Reading Stanford CS336

CS336 Lecture 8: Align ZeRO, FSDP, and 3D Parallelism with Hardware Topology

Lecture 8 moves from parallel primitives to system design: ZeRO progressively shards optimizer state, gradients, and parameters; TP, PP, SP, and EP split width, depth, sequence, and experts. Their composition must follow topology and dynamic activation memory.

#cs336#distributed-training#fsdp#zero#parallelism