🌏 中文版
Video status: recordings require login or course authorization. Sources
This guide follows CMU 10-423 Spring 2026 public slides. Worked examples are original teaching additions.
Three methods address different costs
FlashAttention reduces attention memory traffic. KV caching avoids recomputing historical keys and values. Speculative decoding uses a cheap draft model to propose tokens for target-model verification. Review attention and decoding. We will calculate blockwise normalization and acceptance probabilities.
Ordinary attention materializes n² score/weight pairs. FlashAttention tiles computation into fast local memory and maintains online normalization without writing the full matrix to HBM. It computes exact attention mathematically, subject to floating-point differences. It is not sparse attention and does not eliminate all quadratic arithmetic.
Blockwise readout preserves the answer
Our original one-query example has logits [0,log 3] and values [2,6], giving full-softmax output five. After the first block, running maximum m=0, denominator l=1, and unnormalized weighted sum o=2. The second block changes the maximum to log 3. Rescale old accumulators by exp(−log 3)=1/3: l'=1/3+1=4/3 and o'=2/3+6=20/3. Their ratio is five.
Updating the maximum requires rescaling both accumulators. Normalizing each block independently and simply combining outputs is incorrect. Real kernels add tiling, parallel scheduling, and backward strategies to this online-normalization principle.
Prefill, decode, and cache memory
Prefill processes the prompt; decode adds tokens step by step. Cached K/V let each new query read history without recomputing historical projections. For an original configuration with two layers, four KV heads, head dimension eight, length ten, and FP16, cache bytes are 2×2×4×8×10×2=2560, excluding other tensors. Doubling length doubles this cache.
GQA reduces KV-head count rather than context length. PagedAttention addresses allocation, sharing, and fragmentation, a different problem from FlashAttention's IO. Decode bottlenecks depend on batches, models, and hardware.
Why draft proposals require correction
Use an original two-token vocabulary a/b. Target p=[0.6,0.4], draft q=[0.8,0.2]. A draft a is accepted with min(1,p(a)/q(a))=0.75; b is always accepted. Accepted masses are 0.6 and 0.2, leaving rejection mass 0.2.
After rejection, sample from normalized max(p−q,0). Here only b has positive residual, so all rejected mass becomes b. Final output probabilities are [0.6,0.4], matching the target. Resampling from q after rejection would break correction. This is the ideal sampling argument; greedy variants use different verification rules.
Check your understanding
Identical draft and target distributions accept all proposals, but speed still depends on draft cost and verification overhead. For q=[0.2,0.8], acceptance rates are one for a and 0.5 for b; rejected mass goes to a, again recovering p. Temperature/top-p processing must be reflected in the actual p/q distributions, not replaced with raw-logit probabilities. Verify distributional or greedy equivalence alongside latency and throughput. Continue to long context.
Course video sources
The official Spring 2026 course site directs recordings to SCS Panopto. On 2026-10-12, anonymous retrieval returned an unloaded folder and a sign-in prompt, without playable classroom content. This establishes an access limit, not the absence of recordings. This guide uses public slides and linked primary papers; examples are original teaching additions, not a transcript. Official recording folder.
References
Loading...