Reading Stanford CS329Z Week 2: Tell Workflows from Agents Apart, Then Hand-Build Your First RAG

🌏 中文版

Week 2 is deliberately sequenced. Monday (Sep 28, LLMs for Builders) assigns Anthropic's Building Effective Agents (2024): learn when something should not become an agent before you start building, so you don't over-engineer on day one. Wednesday (Sep 30, RAG) assigns Lewis et al.'s Retrieval-Augmented Generation (NeurIPS 2020): the complete recipe for a first compound system, with an in-class hands-on building a RAG pipeline from scratch. Together they are the blueprint for HW1 Part A.

First, the split: workflows are not agents

Anthropic's first cut is definitional. Workflows run LLMs and tools along predefined code paths; agents let the LLM direct its own process and tool use. Both are agentic systems, but the selection logic differs completely: predictable tasks on fixed paths call for workflows, unpredictable step counts with model judgment call for agents. And most applications need neither — a single LLM call with retrieval and examples is usually enough. That paragraph is the post's brake pedal; every pattern below must answer to it.

Five patterns: the selection toolkit

The building block is the augmented LLM: a model that writes its own search queries, picks tools, and decides what to remember — capabilities tailored to your case with an easy, well-documented interface. Up from there, five patterns in rising complexity:

Prompt chaining: each call processes the previous call's output, with programmatic gates in between. Use it when tasks decompose cleanly; you trade latency for accuracy.

Routing: classify the input, then send it down a specialized path. Use it when distinct inputs deserve distinct prompts — with the precondition that classification itself is accurate.

Parallelization: independent subtasks at once (sectioning), or the same task several times for a vote (voting). Use it for speed or for confidence; complex tasks with multiple considerations usually do better with one focused call per consideration.

Orchestrator-workers: a central LLM decomposes on the spot, delegates, and synthesizes. It differs from parallelization in that subtasks aren't predefined. Fits tasks whose decomposition varies with input, like multi-file code changes.

Evaluator-optimizer: one call generates, another critiques, in a refinement loop. Use it with clear grading criteria where iteration measurably helps — literary translation is the canonical example.

Agents come last: tool loops driven by environment feedback, with checkpoints and iteration caps. For open-ended problems with unpredictable step counts where you trust the model's judgment. Costly, with compounding errors — sandbox plus guardrails as standard kit.

Framework advice: call the API directly first

Anthropic's framework view is blunt: frameworks simplify model calls, tool definitions, and chaining, but each abstraction layer is a debugging blind spot — and an invitation to add complexity you don't need. Start with raw LLM APIs; the same patterns take a few lines of code. If you adopt a framework, understand what's underneath; wrong assumptions about internals are a top customer error source. Appendix 2 applies this to tool definitions: formats should be easy to write (diffs are harder than full rewrites, JSON escaping is costlier than markdown), models need tokens to think before committing, and definitions deserve junior-developer-grade docstrings. On SWE-bench the team spent more time tuning tools than the overall prompt — relative paths broke the model after directory changes, mandating absolute paths fixed it outright. That is ACI (agent-computer interface): whatever effort goes into HCI, tool interfaces deserve the same.

Further reading: context is a budget, not a bucket

Anthropic's Effective Context Engineering for AI Agents widens the unit of design beyond prompt wording. Every inference sees a finite state assembled from system instructions, tool definitions, external data, message history, and tool results; the goal is the smallest set of high-signal tokens that still produces the desired behavior. Static context should stay crisp: tools with clear, non-overlapping contracts and token-efficient outputs, plus a few diverse canonical examples rather than an edge-case catalog.

Dynamic context should arrive just in time. Keep lightweight references such as paths, saved queries, and URLs, then progressively disclose only what the next decision needs. Long tasks add three distinct controls: compaction for an overgrown trace, structured notes for durable state outside the window, and subagents for isolating deep exploration. None is free: compaction can erase details whose value appears later, while autonomous retrieval adds latency and can send the agent down dead ends. This is first-party applied guidance, not a benchmark paper; it supplies a design checklist rather than a promised accuracy gain.

The Week 2 connection is operational. In the RAG exercise, do not stop at “retrieve top-k and paste it.” Hold the retriever fixed, vary top-k, chunk length, tool-result shape, and history pruning, then measure answer quality, tokens, and latency. A second variant can expose only source identifiers first and let the agent fetch full passages on demand.

RAG: the recipe for a first compound system

Last week argued the systems era is here; this week hands over the recipe. The RAG paper's problem statement is precise: big models store knowledge in parameters and fine-tune well, yet remain weak at retrieving and manipulating that knowledge — with provenance and knowledge updates unsolved. The fix is parametric plus non-parametric memory: a seq2seq model (BART) over a dense Wikipedia index, reached through a neural retriever (DPR).

Two formulations: RAG-Sequence conditions the whole generation on one retrieved set, RAG-Token may consult different passages per token. Jeopardy generation shows the difference best: answers often fuse two unrelated facts, so the Token variant reads one document for the first half and switches for the second.

The training setup deserves its own note: retriever and generator train together with no labeled retrieval answers, freezing the document encoder and tuning only the query encoder plus the generator. Retrieval is a DPR bi-encoder — separate BERT encoders for documents and queries, inner-product similarity, top-K via approximate MIPS. Generation is BART-large with input and retrieved passages simply concatenated.

The experiments' breadth is why the course assigns this paper: one architecture across four task families. Open-domain QA (NQ, TriviaQA, WebQuestions, CuratedTrec) set the best known scores — with generative answers beating extractive predecessors. Documents hinting at the answer without quoting it can still contribute, and RAG answers correctly even with nothing relevant retrieved in about one case in nine on NQ, where extractive models score zero. Abstractive QA (MS-MARCO) beats BART by over two BLEU points. Jeopardy generation went to blind human judges: RAG more factual than BART in forty-three percent of pairs against seven. FEVER verification: the top retrieved document is gold evidence seventy percent of the time, rising to ninety within the top ten.

The ablations answer questions HW1 will raise: freezing the retriever costs performance on every task, and swapping dense retrieval for BM25 loses everywhere except entity-dense FEVER, where word overlap still rules. Training retrieves five to ten documents; at test time the Sequence variant keeps improving with more, the Token variant plateaus at ten.

Knowledge updates are equally direct: swap the index file. The authors indexed two Wikipedia vintages and quizzed world leaders — matched years answer around seventy percent right, mismatched years collapse toward ten, with no retraining. That is the mechanism behind Week 1's "dynamic" argument. Related work in one line: REALM and ORQA stayed extractive; RAG is the first general generative recipe for hybrid memory.

For the course, RAG is the cheapest compound-system template — Week 1's three design questions (control logic, resource allocation, end-to-end optimization) finally take a buildable shape.

Further reading: ColBERT keeps fine-grained matching until the end

ColBERT targets the gap between two retrieval extremes. A bi-encoder compresses each query and passage into one vector, making offline indexing easy but discarding fine-grained matches. A cross-encoder jointly processes every query–passage pair, preserving full interaction but paying a Transformer forward pass per candidate. ColBERT encodes query and document separately while retaining a contextualized vector per token. At scoring time, each query token takes its maximum similarity to any document token, and those MaxSim values are summed.

Document token vectors can be precomputed, so the same mechanism supports reranking and full-collection vector search. On the paper's MS MARCO setup, ColBERT reranking reached 34.9 MRR@10 at 61 ms, versus 34.7 at 10,700 ms for the cited BERT-base baseline in the same table. Its end-to-end run over 8.8 million passages reported 36.0 MRR@10, 96.8 Recall@1000, and 458 ms latency. Read those as a 2020 result under the paper's dataset, hardware, and implementation—not as a latency promise for a current deployment. The cost has not vanished either: storing many token vectors per passage makes the index substantially heavier.

The paper measures passage ranking, not downstream RAG factuality or answer quality. That boundary suggests the exercise: compare BM25, a single-vector dense retriever, and ColBERT on the same queries; measure Recall@k or MRR first, then feed identical-sized result sets to the generator and test whether better ranking actually becomes better answers.

What to do: modify last week's RAG

What to do: take the two-stage RAG you hand-built in Week 1 and rewrite it with one of this week's five patterns. The smoothest pick is evaluator-optimizer: add a second LLM call checking the first call's output against the retrieved passages (exactly the example from the compound-AI post). Measure three things: accuracy delta, latency cost, which examples got fixed and which broke. Write the conclusion as one selection sentence: is this task worth an extra call? That is the question Anthropic wants answered before every complexity increase.

Where it sits in the course

Week 2 opens HW1 Part A: Wednesday's hands-on RAG is the assignment's foundation. Week 3's tool use (with MCP entering there) connects tools, Week 4's ReAct fixes the loop's shape — the three Part A puzzle pieces. Reading the Anthropic piece, keep its brake pedal in mind: prove the simple version insufficient before adding each pattern.

This week's course material

• Mon 9/28 LLMs for Builders: anchor reading Anthropic, Building Effective Agents, plus Rajasekaran et al., Effective Context Engineering for AI Agents (Anthropic, 2025); both are covered above.
• Wed 9/30 RAG: anchor reading Lewis et al., RAG, plus Khattab et al., ColBERT; both are covered above.
• Course schedule: CS329Z site

Update log

• 2026-09-12: Added substantive guided readings of Effective Context Engineering and ColBERT.

References

• On this site: Reading Stanford CS329Z Week 1: stop tuning only the model, Stanford CS329Z course guide
• Course: CS329Z schedule
• Sources: Anthropic, Building Effective Agents (2024), Anthropic, Effective Context Engineering for AI Agents (2025), Lewis et al., Retrieval-Augmented Generation, NeurIPS 2020, Khattab & Zaharia, ColBERT, SIGIR 2020
• Tools: litellm docs, MCP specification