CS149 L3: Fast Processors, Slow Data — Latency vs. Bandwidth, and How ISPC Separates Abstraction from Implementation

🌏 中文版

This post is based on the Fall 2025 edition of CS149. It is part 3 of the Reading Stanford CS149 series and covers Lecture 3 (September 30), Modern Multi-Core Architecture (Part II) + ISPC Programming Abstractions. The official slide PDF has 56 pages.

Fall 2025 recordings live only on Stanford's Canvas. The course home page points to the public 2023 recordings instead; this lecture matches 2023 Lecture 3, which has the same title. This post follows the 2025 slides and treats the video as a listening supplement. The course as a whole is A3 (enough for self-study); the gaps are listed in the series overview.

The lecture has two halves that look unrelated. The first is about memory bandwidth; the second is about the ISPC programming model. They share one premise. The previous lecture's three forms of hardware parallelism (multi-core, SIMD, hardware multithreading) made processors enormously capable. The next questions are whether data can arrive fast enough, and how a programmer should describe parallel work.

The lecture opens by finishing the last one

Slide 2 says that hardware multithreading, at the end of L2, did not get covered, so this lecture starts by going through those L2 slides. That material is in the previous post and is not repeated here.

A thought experiment: is element-wise multiply a good fit for a GPU?

Slide 3 poses a question. Take two vectors A and B with millions of elements each, and multiply them element by element into C. Each element needs a load of A[i], a load of B[i], a multiply, and a store to C[i]. The work is fully independent and massively parallel, so it looks like a natural fit for a throughput-oriented processor.

The answer waits until after the latency and bandwidth discussion. For now, keep slide 4's hardware numbers in mind. A V100 has 80 SMs with 64 fp32 ALUs each, for 5,120 ALUs, and a 900 GB/s memory interface. The slide asks you to think about supplying all those ALUs with data every clock.

(GPU architecture gets its own post at order 9. The V100 appears here only to do the bandwidth arithmetic.)

Latency and bandwidth are different things

Slides 6–16 use three everyday analogies to separate two words people often blur:

• Latency: how long one thing takes from start to finish.
• Throughput / bandwidth: how many things finish per unit time. Memory bandwidth is defined as the rate at which the memory system can provide data to a processor.

The highway. San Francisco to Stanford is about 50 km. At 100 km/h, one car takes half an hour. If only one car may be on the road at a time, throughput is 2 cars per hour. You can raise throughput by driving faster (lower latency) or building more lanes (more resources). Or you can let cars follow each other 1 km apart. Latency stays the same, and throughput jumps from 2 cars an hour to 100.

Laundry. Washing takes 45 minutes, drying 60, folding 15, so one load takes 2 hours. Buying a second washer and dryer doubles throughput, but it also doubles the hardware. Pipelining is the alternative: start washing the second load while the first one dries. One washer and one dryer now finish a load every hour, and each load still takes 2 hours.

Two pipes in series. One carries 100 liters per second, the other 50. Connected, the system carries at most 50. The narrowest stage sets the throughput of the whole system.

Put together, the intuition is this: latency can be hidden by keeping many things in flight, but the bandwidth ceiling cannot be hidden.

Back to the processor: the core is waiting for data

Slides 17–20 turn the pipe story into a processor. Each thread repeats three dependent instructions: load 64 bytes, then two adds. The processor:

• does one math operation per clock
• can issue loads in parallel with math
• receives 8 bytes per clock from memory
• has enough hardware threads to hide memory latency

So every 2 clocks of math needs 64 bytes, but memory needs 8 clocks to deliver 64 bytes. The slides' timeline shows it: once the number of outstanding loads hits its limit, the core stalls.

Work the numbers yourself. In steady state, the core does 2 clocks of math out of every 8, so utilization is about one quarter. Slide 20 asks you to convince yourself of something: that utilization depends only on the ratio of instruction throughput to memory throughput. It does not depend on how long memory latency is or how many requests can be outstanding. Memory is transferring data 100% of the time; it simply cannot go faster. This is memory bandwidth-bound execution.

Back to vector multiply

Slide 22 gives the answer. Each multiply reads two floats and writes one, for 12 bytes. At 1.6 GHz a V100 can do 5,120 fp32 multiplies per clock. Keeping every ALU busy would take about 98 TB/s of bandwidth. The card has 900 GB/s.

The result is under 1% GPU efficiency. The slide adds that this is still faster than an eight-core Xeon on a 76 GB/s memory bus, which reaches about 3% efficiency on the same computation.

So the answer to the thought experiment: however much parallelism the program has, bandwidth caps it.

Bandwidth is the critical resource

Slides 23–24 state the conclusion plainly. If processors request data at too high a rate, the memory system cannot keep up. Overcoming bandwidth limits is often the most important challenge for anyone targeting modern throughput-optimized systems.

The slides list what performant parallel programs do:

1. Organize computation to fetch data from memory less often: reuse data the same thread already loaded (temporal locality), or share data across threads.
2. Prefer extra arithmetic over storing and reloading values. "The math is free."
3. In short, access memory infrequently if you want to use a modern processor well.

Order 7 in this series (L6, locality and communication) turns this into a formal discussion of arithmetic intensity. For now, take one habit away: when you see a program, first count how many operations it does per byte it moves.

Something you can do tonight: pick a loop you know well, such as y = a*x + y. Count the bytes it reads and writes per element and the floating-point operations it does. Look up your machine's memory bandwidth and estimate its best possible FLOPS. PA1 Program 5, in the next post, is exactly this problem.

Aside: how does a multiply finish in one clock?

Slide 25 answers a common student question. "One operation per clock" refers to instruction throughput, not latency. In a four-stage pipeline (fetch, decode, execute, write back), each instruction takes 4 clocks, but a new one enters every clock, so throughput is one per clock. The slide notes that modern CPU pipelines can be as deep as about 20 stages, and that back-to-back dependent instructions need care to stay correct. This is the laundry pipeline again.

Second half: abstraction vs. implementation

Slide 26 announces the theme: conflating the semantics (meaning) of a programming abstraction with the details of its implementation is a common cause of confusion in this course.

Slide 27 separates the two layers:

The slide's goal for students: once you know how a programming model is implemented, you should be able to trace in your head what each part of the parallel computer does at each step.

ISPC is the worked example.

ISPC: think in SPMD, implement in SIMD

ISPC is the Intel SPMD Program Compiler. Slide 29 recommends a long read by its author Matt Pharr, The Story of ISPC.

The abstraction: a gang of program instances

Slides 30–36 reuse the sinx() example from L2 (sin computed with a Taylor expansion). When C++ main() calls an ISPC function:

1. The call spawns a gang of programCount program instances.
2. All instances run the same ISPC code at once, each with a different programIndex.
3. When the function returns, every instance has completed, and control goes back to sequential C++.

That is SPMD (single program, multiple data): define one function, run many instances of it in parallel on different data. Slide 50's summary diagram goes: single thread of control → call SPMD function → multiple logical threads of control → return → single thread of control again.

The implementation: SIMD instructions

Slide 49 explains the implementation. The ISPC compiler emits vector instructions (for example AVX2 or ARM NEON) that carry out the logic of an entire gang. For conditional control flow, the compiler masks vector lanes — the same thing PA1 has you do by hand.

So the programmer imagines programCount independent instruction streams. The machine actually runs one thread executing a sequence of SIMD instructions.

Same abstraction, different work assignments

Slides 34–40 compare two versions:

• Interleaved: instance 0 handles elements 0, 8, 16…; instance 1 handles 1, 9, 17…
• Blocked: each instance gets one contiguous chunk.

Both compute the same answer, but they are implemented very differently. Slides 39–40 show which elements the gang touches at each step. With interleaving, the 8 instances read 8 contiguous values, which a single packed vector load (vmovaps) can handle. With blocking, the 8 values are not contiguous and need a gather instruction (vgatherdps), which the slides describe as more complex and more costly. That is abstraction vs. implementation in miniature.

foreach: only say which work is independent

Slides 41–43 introduce foreach. Writing foreach (i = 0 ... N) declares only that the N iterations are independent and can run in any order. It says nothing about which instance runs which iteration. Slide 42 lists four implementations the compiler could choose: everything on instance 0, interleaved, blocked, or dynamic assignment. The point: foreach is the abstraction, and all four are valid implementations.

Slide 43 notes that in many simple cases, foreach lets you write a parallel program almost as if it were sequential.

The abstraction has limits: undefined output

Slides 44–45 test whether you understand the semantics. The first program writes the absolute value of each x twice into y. Each iteration writes to its own locations, so the result is well defined. The second writes to y[i-1] under some conditions and y[i] otherwise, so two iterations can write the same location — the output is undefined. foreach records your claim that iterations are independent. It does not check that claim for you.

Array sum, on slides 46–48, is another trap. If every instance accumulates into a non-uniform variable, you end up with programCount partial sums that cannot be returned to C code expecting a single value, and the compiler reports a type error. The correct version has each instance accumulate a private partial sum, then combines them with the cross-instance primitive reduce_add(). Slide 48 also lists reduce_min, broadcast, and rotate.

Slide 47 includes a self-test: if you understand why a particular AVX-intrinsics function correctly implements the semantics of this ISPC function, you have a good command of ISPC.

(How to actually write uniform, programIndex, and friends comes in the next post, PA1 Program 3. Here you only need to know which layer they belong to.)

One layer still missing: multiple cores

Slide 51 is a reminder: a gang runs as SIMD instructions in one thread on one CPU core. All the code so far used only one of the four cores on a myth machine. ISPC has another abstraction for multi-core execution, the task, and the slide leaves it for students to read up on during Assignment 1.

The cost of a low-level language, and higher-level options

Slides 52–55 end with a design question. ISPC is a low-level language. By exposing programIndex and programCount, it lets you control exactly what each instance does and which data it touches. The cost is that you can also write programs with undefined output, or programs that are correct only for one specific programCount.

Remove programIndex and programCount and allow only foreach, and you barely need to think about program instances at all. Go one step further, drop array indexing entirely, and only allow map(f, x) over a collection. The slides point out that this should feel familiar to NumPy and PyTorch programmers, and promise much more on it later.

Slide 56's summary is three sentences: programming models are a way to think about how parallel programs are organized; they provide abstractions that permit multiple valid implementations; keep thinking about abstraction vs. implementation for the rest of the course.

Two habits to take away

1. Count bandwidth before you count parallelism. A program with endless parallelism is still stuck on memory if it does one operation per byte.
2. When reading parallel code, know which layer you are reading. "What answer does this compute?" and "How does it run on the machine?" are separate questions. foreach answers the first; the compiler answers the second.

Further reading: to see bandwidth limits at the scale of large-model training, continue with this site's CS336 guide to GPUs and TPUs.

Series navigation: previous L2: A Modern Multi-Core Processor | next PA1 + Written 1: Performance on a Quad-Core CPU | series overview

References

• Stanford CS149 Fall 2025 course home page
• Lecture 3 page (slide-by-slide)
• Lecture 3 slides PDF: Multi-Core Architecture, Part II + Parallel Programming Abstractions
• 2023 Lecture 3 recording: Multi-core Arch Part II + ISPC Programming Abstractions
• CS149 2023 public recordings playlist
• ISPC official site
• Matt Pharr: The Story of ISPC