CME295 2026 Lecture 5 (Pre-Lecture Edition): LLM Systems, or How the Same Model Runs Several Times Faster

🌏 中文版

Pre-lecture edition: This post was written on September 29, 2026, before Lecture 5 of the 2026 edition (October 30, 2026) has taken place. It is based on the topic list in the 2026 syllabus, the parts already covered in the 2025 slides, and the original papers. It will be revised against the video and slides once they are posted.

This post covers Lecture 5, "LLM systems," of the 2026 edition of Stanford's CME295. The lecture did not exist in 2025. Its material was spread across the last ~40 slides of the 2025 Lecture 3 deck (inference speedups) and ~50 slides in the middle of the Lecture 4 deck (training optimizations). Our guides to Lecture 3 and Lecture 4 each drew a map of these topics and deferred the details to this post.

If you have used any LLM chat interface, you know the pattern: you send a question, wait a moment for the first word, and then words stream out at a steady pace. Those two waits are different kinds of work with different bottlenecks. The topic list for this lecture looks scattered, running from multi-GPU training to hardware choices, but most of it reduces to one idea: moving data around a GPU often costs more than the computation itself.

Where this lecture sits in the 2026 syllabus

The 2026 syllabus schedules Lecture 5 for October 30, right after the October 23 midterm and before Lecture 6 on AI Agents. It lists exactly seven topics:

1. Distributed training
2. Inference optimizations
3. KV caching
4. Speculative decoding
5. Efficient kernels
6. Flash Attention
7. Hardware trade-offs

Two clarifications up front. First, quantization and mixed precision were covered in the 2025 Lecture 4, but the 2026 syllabus does not list them under Lecture 5, so this post does not treat them as part of it. Second, MHA/MQA/GQA are listed under Lecture 2 in 2026, so here they only appear as a way to shrink the KV cache.

The sections below follow the syllabus order. Each one says which content comes from the 2025 slides and which comes from papers or other courses.

flowchart TB
  subgraph TR["Training: the model doesn't fit, the compute doesn't finish"]
    DT["Distributed training<br/>DP · ZeRO · TP · PP"]
  end
  subgraph INF["Inference: one token at a time"]
    PF["prefill<br/>whole prompt in parallel<br/>often compute-bound"]
    DC["decode<br/>reads all the weights every step<br/>often memory-bandwidth-bound"]
    PF --> DC
    DC --> KV["KV caching<br/>don't recompute past K, V"]
    DC --> SD["speculative decoding<br/>verify several tokens at once"]
  end
  KN["Efficient kernels · FlashAttention<br/>fewer HBM reads/writes"] --> DT
  KN --> PF
  HW["Hardware trade-offs<br/>compute · bandwidth · capacity · interconnect"] -.-> TR
  HW -.-> INF

First, one ledger: compute fast, or move data fast

This section is the key to everything that follows. The 2025 slides don't frame it this way; it's the framework used in CS336 Lecture 2 and CS336 Lecture 10, which I borrow to tie the lecture together.

A GPU has two key specs: how many operations it can do per second, and how much data it can move from memory per second. According to NVIDIA's H100 spec page (checked 2026-09-29), the H100 SXM lists 1,979 TFLOPS of BF16 compute. That figure is footnoted "with sparsity"; dense matrices get roughly half, about 989 TFLOPS. Memory bandwidth is 3.35 TB/s. Divide one by the other and you get about 295: the GPU has to do around 295 operations for every byte it moves before compute becomes the limit.

That ratio is a threshold on arithmetic intensity. Work below it leaves the GPU waiting for data with its compute idle; only work above it is truly compute-bound.

H100 threshold ≈ 989e12 FLOP/s ÷ 3.35e12 byte/s ≈ 295 FLOP/byte

Decoding one token (batch = 1, BF16 weights):
  each parameter: read 2 bytes, do 1 multiply + 1 add = 2 FLOP
  arithmetic intensity ≈ 2 FLOP / 2 byte = 1 FLOP/byte

With batch = B, one read of the weights is shared by B requests:
  arithmetic intensity ≈ B FLOP/byte

Keep this ledger in mind. Every topic below answers one of three questions: does the data fit? Can it be moved fast enough? Can we do more work per move?

Distributed training: how to split when one GPU isn't enough

2025 source: Lecture 4 slides, pp. 32–44.

The Lecture 4 guide already listed what a training step has to store: parameters, activations, gradients, and Adam's two optimizer states. The ZeRO paper turns that into numbers: with mixed-precision Adam, each parameter needs 16 bytes. That's 2 for FP16 parameters, 2 for gradients, and 12 for the FP32 parameter copy plus two moment estimates. The paper's example is a 7.5B-parameter model, whose "model states" alone take about 120 GB before counting activations. That doesn't fit on an 80GB H100.

The 2025 slides split the solutions into two routes:

Plain data parallelism:       2Ψ + 2Ψ + 12Ψ        = 16Ψ bytes
ZeRO-1 (shard optimizer):     2Ψ + 2Ψ + 12Ψ / N_d
ZeRO-2 (+ shard gradients):   2Ψ + 14Ψ / N_d
ZeRO-3 (+ shard parameters):  16Ψ / N_d

PyTorch's built-in FSDP (Fully Sharded Data Parallel) takes the same fully sharded route as ZeRO-3. Its paper reports performance comparable to plain data parallelism while supporting much larger models.

For model parallelism, the slides give only the names and one suggested reading, Hugging Face's Ultra-Scale Playbook. The two most common forms:

• Tensor parallelism (TP): split the matrix multiplications inside a layer across GPUs, as in Megatron-LM. Every layer exchanges results, so communication is very frequent.
• Pipeline parallelism (PP): put different layers on different GPUs and let data flow through like an assembly line, as in GPipe. Less communication, but at the start and end of the pipeline some GPUs sit idle waiting.

For how these are built from communication primitives like all-reduce and all-gather, see CS336 Lecture 7; for how to combine them into 3D parallelism that respects network topology, see CS336 Lecture 8.

Inference optimizations: prefill and decode are two different jobs

2025 source: Lecture 3 slides, pp. 86–89 and p. 124 (the classification framework). The slides don't use the terms prefill/decode; those come from the inference-systems literature.

Back to the opening experience. When you send a prompt, the model first reads the whole thing and computes keys and values at every position. This is prefill. All prompt tokens are known, so the work parallelizes, the matrices are large, arithmetic intensity is high, and the GPU's compute is well used. Most of your wait for the first word (TTFT, time to first token) is spent here.

Then comes decode: each step produces one token, and the next step can't start until it's done. Every step reads the entire set of weights from HBM to compute a single token. As the ledger showed, arithmetic intensity here is about 1, so the GPU spends most of its time waiting for data. How fast words stream out depends on this phase.

The 2025 slides split inference optimizations into two broad classes, and the final slide maps each technique back:

One cell deserves attention: the slides place speculative decoding in the "exact" class. It really doesn't change the output distribution, only the order of computation; more on that below.

Another technique the slides don't mention but nearly every inference engine uses is continuous batching. Decode's arithmetic intensity is proportional to batch size, so packing more requests together pays off. The catch is that requests have different lengths, and the traditional approach waits for a whole batch to finish before starting the next. Orca (OSDI 2022) schedules at the granularity of a single step instead: as soon as one request finishes, a new one joins at the next step.

KV caching: trading memory for recomputation

2025 source: Lecture 3 slides, pp. 91–115.

The slides walk through "a cute teddy bear is reading" one word at a time. To produce the 6th token, the model attends to the previous 5 and needs their keys and values; to produce the 7th, it needs them again. Those old K and V never change, so the slide's idea is "Keep keys and values in a cache": compute once, store, and read back afterward.

This is a classic memory-for-compute trade. The price is that the KV cache grows with the conversation, and every concurrent request has its own copy.

KV cache bytes = 2 × layers × KV heads × head dimension
                 × sequence length × batch × bytes per value
(the leading 2 = one copy each for K and V)

Per token:   2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB
8,192 tokens: 128 KiB × 8,192 = 1 GiB (single request)

A larger KV cache means fewer concurrent requests per GPU, smaller batches, and even lower decode arithmetic intensity. The 2025 slides list three ways to shrink or manage it:

• Store fewer heads (GQA/MQA): several query heads share one set of K and V. MQA shares a single set across all heads; GQA shares within groups. Llama 3 pairs 32 query heads with 8 KV heads, cutting the KV cache to a quarter. The 2026 syllabus puts this in Lecture 2, which our Lecture 2 guide covers.
• Store a compressed version (latent attention): DeepSeek-V2's Multi-head Latent Attention stores a low-dimensional latent vector instead of full K and V, reconstructing them when needed. The paper reports that, compared with DeepSeek 67B, it reduces the KV cache by 93.3% and raises maximum generation throughput to 5.76 times.
• Manage memory better (PagedAttention): the vLLM paper measured that existing systems, which reserve one contiguous chunk per request, use only 20.4% to 38.2% of KV cache memory for actual token states; the rest is lost to fragmentation and over-reservation. PagedAttention borrows paging from operating systems, splitting the KV cache into fixed-size blocks that need not be contiguous, which limits waste to at most one block per request. The paper reports 2–4× higher throughput than the systems of the time.

PagedAttention has a side benefit: requests that share a prefix (for example, the same system prompt) can share the same blocks. For vLLM's implementation, see our vLLM deep dive.

Speculative decoding: use the idle GPU to guess ahead

2025 source: Lecture 3 slides, pp. 116–122. The full acceptance rule is in the Lecture 3 guide and isn't repeated here.

During decode the GPU's compute is mostly idle, so can we make it do more per step? The observation in Chen et al. is that having the large model verify several tokens at once takes about as long as having it generate one. Both require reading the weights, and reading the weights is the dominant cost.

So the procedure is:

1. A small draft model quickly guesses the next k tokens
2. The large target model computes probabilities for all k positions in one parallel pass
3. The acceptance rule checks the guesses left to right, accepts up to the first rejection, and resamples one token at that position

The slides' example has the draft model continue "[BOS] my teddy bear" with "is cute and smart," which the target model verifies in one pass. The acceptance rule is designed so that, as Leviathan et al. and Chen et al. each prove, the final output distribution is identical to sampling from the large model alone. That's why the slides file it under "exact."

Expected tokens per round = (1 − α^(γ+1)) / (1 − α)

Reported speedups: 2–3× on T5-XXL in Leviathan et al., and 2–2.5× on the 70B-parameter Chinchilla in Chen et al.

The slides end with a variant, multi-token prediction (MTP): instead of maintaining a separate draft model, train several prediction heads on the same model to predict multiple upcoming tokens, so draft and target are the same model. Gloeckle et al. report that models trained with 4-token prediction run up to 3× faster at inference. The same line of thinking includes Medusa (extra decoding heads) and EAGLE (prediction at the feature level); EAGLE reports a 2.7–3.5× latency speedup on LLaMA2-Chat 70B.

It isn't always worth it. vLLM's speculative decoding docs (checked 2026-09-29) state up front that it is meant to "reduce inter-token latency under medium-to-low QPS, memory-bound workloads." Under heavy traffic, batches are already large and the GPU is already busy computing, so the extra guesses compete for compute.

Efficient kernels: fusing several steps into one

2025 source: none as a standalone section. The 2025 edition covered the same principle only within FlashAttention; this is a newly listed topic in 2026. What follows is based on CS336 Lecture 5, CS336 Lecture 6, and official docs.

On a GPU, a kernel is a function launched for the GPU to execute. If you write y = gelu(x @ W + b) in PyTorch, the most direct execution is three kernels: the matmul writes its result to HBM, the add reads it back and writes again, and GeLU reads and writes once more. Of those three round trips, only the first read and the last write are actually necessary.

Kernel fusion merges those steps into one kernel so intermediate results stay in fast on-chip memory instead of going back to HBM. For low-intensity elementwise operations (adds, activations, normalization, softmax), this is usually the biggest source of speedup.

Writing fused kernels used to require CUDA C++. There are easier routes now:

• Triton lets you write GPU kernels in Python syntax; the second official tutorial is a fused softmax
• PyTorch's torch.compile generates fused kernels automatically. In the TorchInductor docs, generated kernels carry names like triton_poi_fused_cat_155, which says on its face that it's a fused Triton kernel

The most common trap in kernel work is optimizing before measuring. That is exactly the key point of CS336 Lecture 6: measure first, then use a profiler to find where the time actually goes.

FlashAttention: kernel optimization applied to attention

2025 source: Lecture 4 slides, pp. 45–71. The Lecture 4 guide already quoted the paper's numbers (more FLOPs, but HBM reads/writes drop from 40.3 GB to 4.4 GB and runtime from 41.7 ms to 7.3 ms). This section fills in the mechanism.

The slides draw the problem with standard attention step by step: load Q and K from HBM, compute the score matrix S, write S to HBM; read S back, compute the softmax P, write P to HBM; read P and V, compute the output O. S and P are both sequence-length × sequence-length matrices, so for long sequences these round trips dominate.

FlashAttention rests on two ideas, each given a slide:

1. Tiling: split Q, K, V into blocks, move one block at a time into SRAM, compute that block's output there, and write it back. S and P never touch HBM.
2. Recompute in the backward pass: normally S and P are stored for the backward pass. FlashAttention doesn't store them; it recomputes them with the same tiling. The slide puts it as "Sometimes, it is better to recompute instead of storing."

Tiling has a catch: softmax needs the row's maximum and sum before it can normalize, but you only see one block at a time. Slide 62 says "No need to compute the full S = QKᵀ before applying softmax" and shows the softmax of a full row as per-block softmaxes each multiplied by a coefficient α. The technique is often called online softmax: with each new block, update the running maximum and sum, and rescale what has been accumulated so far.

S_j   = Q · K_jᵀ                          # scores for this block
m_new = max(m, rowmax(S_j))
ℓ_new = e^(m − m_new) · ℓ + rowsum(e^(S_j − m_new))
O_new = e^(m − m_new) · O + e^(S_j − m_new) · V_j

After all blocks: output = O / ℓ

Two later versions push the same idea closer to the hardware:

The FlashAttention-3 paper notes that v2 reached only 35% utilization on the H100. The same algorithm has to be rewritten for each hardware generation, which leads straight into the last topic.

Hardware trade-offs: there is no free speedup

2025 source: Lecture 4 slides, pp. 36–37 (limited H100 memory) and pp. 72–77 (number formats and "lower precision → faster processing"). "Hardware trade-offs" as a topic is new in the 2026 syllabus; the summary below draws on the sections above and official specs.

Lay out every technique so far and each one trades one resource for another:

The last row ties directly to hardware specs. GPUs inside one machine connect over NVLink, listed at 900 GB/s on the H100 spec page; across machines or over PCIe Gen5 it's 128 GB/s. That's why the most communication-heavy form, tensor parallelism, usually stays within a single machine, while pipeline or data parallelism spans machines.

Choosing hardware means looking at more than one number:

• Compute (FLOPS): how fast prefill and training run
• Memory bandwidth: how fast decode runs
• Memory capacity: whether the model and KV cache fit, and how many requests can be served at once
• Interconnect bandwidth: whether splitting across GPUs pays off

Slide 77 of the 2025 deck also says "Lower precision → Faster processing," the starting point for quantization and mixed precision. The 2026 syllabus doesn't list quantization; whether it shows up under "hardware trade-offs" won't be known until the slides are out.

Connecting back to the models you use

• Using an API: a long prompt delays the first word; that's prefill. How fast words stream is decode. They are optimized differently, so when comparing inference services, look at time to first token and output tokens per second separately.
• Self-hosting: estimate the KV cache first. Plug your model's configuration, expected context length, and concurrency into the formula above, and you'll often find the KV cache, not the weights, is what limits you. vLLM manages it with PagedAttention; our vLLM self-hosting decision works out how GPU utilization drives cost per million tokens.
• Considering speculative decoding: look at your traffic first. It works well at low traffic when single-user latency matters; at high traffic it may not help.
• Training or fine-tuning: if the model doesn't fit, try ZeRO/FSDP first, which needs no model code changes; move to tensor or pipeline parallelism only if it still doesn't fit.

Where the 2025 edition covered this

The 2026 Lecture 5 slides have not been released. Here is how the seven 2026 syllabus topics map onto 2025 slide pages:

Conversely, the number formats and mixed-precision training on 2025 Lecture 4 pp. 72–81, and the quantization used by QLoRA, do not appear on the 2026 Lecture 5 list.

Self-check

Questions 1, 2, 3, and 6 are adapted from Parts III and IV of the 2025 midterm; answers are in the solutions PDF. The 2026 midterm is on October 23, before this lecture. Going by the 2025 split (midterm on Lectures 1–4, final on Lectures 5–8), this lecture would fall under the December 9 final; the course hasn't announced the scope.

1. What is the KV cache for at inference time, and what cost does it save? (Question III.5)
2. Compared with MHA, what is the main way MQA/GQA reduce latency? (Question III.6)
3. What problem is PagedAttention mainly trying to solve? (Question III.3)
4. (My own question) When the draft model in speculative decoding guesses wrong, why is the final output distribution still the same as using the target model alone? How does the speedup change as the draft's acceptance rate drops?
5. (My own question) What extra state does each of ZeRO-1, ZeRO-2, and ZeRO-3 shard? What do you pay for sharding more?
6. Explain the core idea of FlashAttention and give one concrete benefit observed in practice. (Question IV.10)
7. (My own question) Using this post's KV cache formula, compute the KV cache size for Llama 3 8B at 32K context with 4 concurrent requests in BF16. Then explain why decode's arithmetic intensity is roughly equal to the batch size, and what that implies for whether to enable speculative decoding.

Going deeper

• Counting FLOPs, memory, and rooflines yourself: CS336 Lecture 2
• GPU memory hierarchy and the principles behind FlashAttention: CS336 Lecture 5
• Writing Triton kernels: CS336 Lecture 6
• Communication primitives and the three parallelisms: CS336 Lecture 7, CS336 Lecture 8
• The full inference ledger (prefill/decode, quantization, continuous batching): CS336 Lecture 10
• Get a feel for TTFT and speculative decoding with sliders: Learn Inference, an interactive inference-engineering site
• How PagedAttention became a product: vLLM deep dive
• The other new 2026 lectures in this batch: RL with LLMs, AI Agents, Diffusion LLMs

Update plan

Once the slides and video go up on October 30, this post will be revised against the following:

• What each of the seven topics actually covered, and how it differs from this reconstruction from 2025 slides and papers
• Whether "efficient kernels" means kernel fusion and Triton, or something else
• Whether "hardware trade-offs" folds in quantization and number formats
• Whether GQA, latent attention, and PagedAttention are taught here or stay in Lecture 2
• Replacing the "Where the 2025 edition covered this" table with 2026 slide pages, and updating the self-check from the 2026 final

References

• CME 295 2026 syllabus (checked 2026-09-29)
• CME 295 2025 syllabus
• 2025 Lecture 3 slides (PDF) / recording
• 2025 Lecture 4 slides (PDF) / recording
• 2025 midterm / solutions
• NVIDIA H100 Tensor Core GPU specs (checked 2026-09-29)
• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019)
• Zhao et al., PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023)
• Shoeybi et al., Megatron-LM (2019)
• Huang et al., GPipe (2018)
• Hugging Face, The Ultra-Scale Playbook (2025)
• Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022)
• Llama Team, The Llama 3 Herd of Models (2024)
• Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)
• Ainslie et al., GQA (2023)
• DeepSeek-AI, DeepSeek-V2 (2024)
• Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023)
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding (2022)
• Chen et al., Accelerating Large Language Model Decoding with Speculative Sampling (2023)
• Gloeckle et al., Better & Faster Large Language Models via Multi-token Prediction (2024)
• Cai et al., Medusa (2024)
• Li et al., EAGLE (2024)
• vLLM docs: Speculative Decoding (checked 2026-09-29)
• Triton documentation / Fused Softmax tutorial
• PyTorch docs: TorchInductor GPU Profiling
• Dao et al., FlashAttention (2022)
• Dao, FlashAttention-2 (2023)
• Shah et al., FlashAttention-3 (2024)
• Reading Stanford CME295 (series overview)