Testing minGPT Changes: Shapes, Masks, and Positions | CMU 10-423 Assignment 1
Assignment 1 brings sequence modeling and modern attention into code.
ai
1239 postsTesting DDPM: Matching Timestep Coefficients to a Batch | CMU 10-423 Assignment 2
Copying a noising equation is not enough to implement diffusion correctly.
Testing LoRA Fine-Tuning: Gradients, Data, and Evaluation | CMU 10-423 Assignment 3
Use an original numerical test to check gradients, data, and evaluation alongside the original assignment specification.
Testing Q-Former Text-to-Image: Gradients through Frozen Backbones | CMU 10-423 Assignment 4
Use an original numerical test to check gradients, data, and evaluation alongside the original assignment specification.
SFT, RLHF, and DPO: Separate Their Training Signals | CMU 10-423 Topic Guide
A prompt with one demonstrated answer is not the same data as a prompt with a preferred and rejected response. SFT learns demonstrated tokens. A typical RLHF pipeline learns reward and then updates a constrained policy. DPO forms a loss directly from preference pairs and reference-relative ratios.
Long Context and SSMs: Different Ways to Retain History | CMU 10-423 Topic Guide
Long-context evaluation asks whether a model can process, retrieve, and use distant information. An SSM specifies sequence processing through state updates. SSM efficiency does not establish reliable document retrieval, and a large attention window does not guarantee relevant evidence is used.
Modern Transformers: What Position Rotation and KV Sharing Change | CMU 10-423 Lecture 4
Attention needs both order information and a manageable memory footprint during generation.
PEFT and ICL: Change Parameters or Change the Input | CMU 10-423 Topic Guide
Adapting without retraining every weight can mean two different things. LoRA-style PEFT still trains some parameters. ICL supplies examples in context without updating parameters during that inference. The source spans the end of Lecture 9, Lecture 10, and the ICL material in Lecture 11.
Reasoning Models: Spending Compute on Verifiable Answers | CMU 10-423 Lecture 20
Reasoning Models: Spending Compute on Verifiable Answers | CMU 10-423 Lecture 20
Operational Risks and Alignment Theory: Tests and Assumptions | CMU 10-423 Topic Guide
Hallucination, leakage, and prompt injection are observable operational risks. Alignment theory asks which guarantees hold under specific formal assumptions. They require different evidence. Lectures 22 and 26 therefore have separate teaching guides, without turning one theorem into a universal impossibility claim.
RNN Language Models: Sequence Probabilities and Gradient Paths | CMU 10-423 Lecture 1
A sequence such as “I like cats” is not a bag of independent words.
Scaling and MoE: Compute Budgets and Conditional Computation | CMU 10-423 Topic Guide
Scaling laws study model and data allocation under a budget. MoE increases expert capacity while activating selected experts. The former concerns allocation and empirical models; the latter concerns architecture and routing. Both deserve separate checks.
Text-to-Image and VLMs: Generating Images versus Reading Them | CMU 10-423 Topic Guide
“Text plus images” does not specify the task. Text-to-image generation produces an image conditioned on text; a VLM may consume an image and produce an answer. Direction, output objective, and evaluation differ even when both use Transformers.
Transformer, Training, and Decoding: Three Different Questions | CMU 10-423 Topic Guide
How the model reads a history, learns from targets, and chooses its next token are different questions. Attention mixes information; the training objective updates parameters; decoding selects output under a fixed model. Keeping them distinct prevents confusing a sampling change with a model update.
Diffusion and VAEs: Related Objectives, Different Generation Paths | CMU 10-423 Topic Guide
Diffusion and VAEs share probabilistic ideas without sharing the same generation procedure. Diffusion chains noise steps; a VAE approximates a latent posterior with an encoder and reconstructs through a decoder. This guide connects those ideas while directing each mechanism to its own lesson.
Accelerating Transformers on GPUs, and Where LightSeq Finds the Time | CMU 11-868 Lecture 10
Lecture 10 of 11-868 uses Lei Li's own LightSeq and LightSeq2 as the case study and breaks them into four techniques: fuse every small operation outside matrix multiplication into one kernel, rewrite the LayerNorm and Softmax formulas to cut thread synchronizations, store parameters and gradients in FP16 but compute updates in FP32, and reuse memory based on backward-pass dependencies. The slides report 1.4-3.5x training speedups on WMT14 English-German. The official syllabus lists no public recording link; this guide works from slide page numbers and the two papers.
Distributed Training and Data Parallelism, and Where Gradient Sync Costs Come From | CMU 11-868 Lectures 14–15
Lectures 14 and 15 of 11-868 go from the parameter server to PyTorch DDP. They use NCCL's five collectives (Broadcast, Reduce, AllReduce, ReduceScatter, AllGather) as building blocks, show why a ring makes broadcast time nearly independent of GPU count, and split AllReduce into ReduceScatter plus AllGather. The second lecture takes apart DDP's two key designs: bucketing gradients (25 MB by default) and starting synchronization before the backward pass finishes. The official syllabus lists no public recording link; this guide works from slide page numbers and the VLDB 2020 paper.
How a Deep Learning Framework Computes Gradients from a Computation Graph | CMU 11-868 Lecture 5
L05 follows a small sentiment classification network throughout. It expresses computation as a graph, evaluates it in topological order, sends gradients back with the chain rule and vector-Jacobian products, and then takes apart TensorFlow v1's placeholder, variable, operation, and session. One slide is labeled "important for HW2".
FlashAttention: Attention Is Slow Because of Data Movement, Not Math — Tri Dao from FA1 to FA4 | CMU 11-868 Lecture 21
Standard attention writes the N×N score matrix out to HBM and reads it back, and most of its time goes to that traffic. FlashAttention uses tiling plus softmax rescaling so each block finishes inside SRAM, and the backward pass recomputes instead of storing. Tri Dao's guest slides for 11-868 give one set of numbers: the backward pass does 13% more FLOPs, 9x less HBM traffic, and runs 6x faster. FA3 and FA4 follow the same theme: when the hardware changes, the bottleneck moves, and the algorithm has to move with it.
GPU Programming and Acceleration: Threads, Blocks, the Memory Hierarchy, and Tiling | CMU 11-868 Lectures 2–4
CMU 11-868's three GPU lectures answer one question: why does a correct CUDA matmul use only 2.48% of an A100's FP32 compute? L02 covers SMs, warps, and the grid/block/thread hierarchy. L03 covers cudaMalloc, cudaMemcpy, and kernel indexing. L04 uses tiling, coalesced access, and bank-conflict avoidance to bring data closer than global memory, which sits about 500 cycles away.