🌏 中文版
Video status: recordings require login or course authorization. Sources
This guide follows CMU 10-423 Spring 2026 public slides. Worked examples are original teaching additions.
More GPUs do not specify one algorithm
Data parallelism splits examples; tensor parallelism splits operations within a layer; pipeline parallelism splits layers; state sharding divides storage of optimizer state, gradients, and parameters. Review autodiff and Transformers. We will calculate a two-device update and distinguish storage from computation.
Data-parallel devices typically hold identical model replicas, compute local gradients, synchronize, and update consistently. Doubling devices does not automatically halve each replica's weight memory. A model that cannot fit needs another form of splitting or sharding.
Calculate one synchronized step
Our original scalar model is ŷ=wx, initially w=1, with per-example loss 0.5(wx−y)². Device A gets (1,3) and gradient −2. Device B gets (2,1) and gradient two. With one example each, the global mean is zero, so learning rate 0.1 leaves both weights at one.
Independent steps would produce 1.2 and 0.8, breaking replica agreement. A summed collective without correct scaling can also enlarge updates. This canceling example hides that error, so change B's target to two: g_B=0, global gradient=−1, and the correct new weight is 1.1. Forgetting to average would give 1.2.
Unequal batches require weighting
If A has one example and B three, averaging their local mean gradients equally does not recover the four-example mean. Use (g_A+3g_B)/4. Check loss reduction, sampler behavior, synchronization conventions, and accumulation scaling together. Otherwise effective learning rates can depend accidentally on microbatch count.
Dropout, floating-point operations, and collective ordering can prevent bitwise agreement with a single-device run. A meaningful check compares gradients for the same objective within tolerance and verifies consistent replicas.
Splitting models versus splitting state
Tensor parallelism can split output dimensions across devices, with gather or reduce operations when later computation needs combined results. Pipeline stages overlap microbatches but retain dependencies and fill/drain bubbles. Granularity and communication determine speed, not device count alone.
ZeRO progressively shards optimizer states, gradients, and parameters to reduce replicated storage. Adam moments and mixed-precision master weights can consume substantial memory, with bytes depending on implementation. State sharding changes storage and communication; it does not automatically halve every layer's arithmetic.
Check your understanding
One example with mean gradient −2 plus three with mean gradient two yields a global mean of one, not zero. Learning rate 0.1 gives w=0.9. Four-way pure data parallelism does not turn a 10 GB weight replica into 2.5 GB per device. Large optimizer state may favor state sharding; an oversized single layer may require parameter or tensor splitting. Continue to efficient inference.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, anonymous retrieval returned an unloaded folder and a sign-in prompt, without playable classroom content. This establishes an access limit, not the absence of recordings. This guide uses public slides and linked primary papers; examples are original teaching additions, not a transcript. Official recording folder.
References
Loading...