Reading Stanford CS329Z Week 8: Let a Model Judge, Then Guardrail the Agent

🌏 中文版

Week 8 is judges week plus safety week. Monday (11/9) covers LLM-as-a-Judge and eval infrastructure. Wednesday (11/11) covers Agent Safety and Guardrails. The paper video is due Friday, and quarter projects enter final rehearsal.

Open-ended answers have no answer key, and that is the week's starting point. Classic multiple-choice benchmarks measure right versus wrong, like MMLU. Two answers can both be correct while only one is actually useful, and that gap is invisible to classic benchmarks.

Human grading is the gold standard, at gold-standard prices and speed. Every prompt change cannot mean rehiring dozens of graduate students to vote again. The fix is asking a strong model to judge, validated end to end in the MT-Bench paper. Anthropic's eval guide then slots the judge into the full agent-evaluation puzzle. Judges make mistakes too, and Wednesday's safety lecture answers the other question: why do agents that pass grading still fail in production?

Of the three anchor readings, this guide reads the Anthropic eval guide and MT-Bench closely. The safety thread centers on PrivacyLens, plus deployment-time guardrail practice.

Three grader types: pick the right tool before automating

Anthropic sorts graders into three kinds. Code-based graders check deterministic evidence: string matching, unit tests, static analysis, tool-call records. Model-based graders check open-ended quality: rubric scoring, natural-language assertions, pairwise comparison, reference-based grading, multi-judge consensus. Humans serve as the gold standard: expert review, crowdsourcing, spot-check sampling.

The selection rule is pragmatic: wherever a deterministic check works, use it. Reserve model judges for places only they can reach, and reserve humans for calibrating model judges plus periodic sampling. One support task can carry all three grader types: whether the ticket landed resolved in the database, whether the key tools were called by the book, and whether the tone showed empathy. The first two are code-based; only the last one calls a model.

Note the conclusion early: the judge is just one cell in the model-based column, not the whole eval. The wrong grader choice is beyond any judge's rescue.

Pairwise, pointwise, and reference answers

Judge shapes come in three. Pairwise judging places two answers side by side and picks the winner without scoring. Pointwise judging scores a single answer directly, also called single-answer grading. Reference-guided judging treats math and reasoning questions specially: attach the reference solution, then judge.

The tradeoff sits between cost and sensitivity. Pairwise judging is sensitive, while its grading workload grows quadratically with the number of contestants. Pointwise judging scales well, while absolute scores drift with the judge model. MT-Bench validation shows GPT-4 pointwise and pairwise results agreeing closely, with a fairly stable internal rubric.

Math questions are the judge's weak spot: solving a problem does not imply grading one. Wrong answers drag the judge along, and even basic questions can get misjudged. Two mitigations help: let the judge solve the question independently before grading, or attach the reference answer outright. With the reference attached, the failure rate falls to roughly fifteen percent.

Bias: three bad habits of judges

The first bad habit is position bias: a standing preference for whichever answer comes first. The fix is judging twice with positions swapped, counting a win only when an answer wins both rounds, and calling the rest ties. Double the grading cost buys one trustworthy outcome.

The second is verbosity bias: long, watery answers get favored. The paper tests it with a "repetitive list" attack: rephrase an answer's list and paste it back in, adding zero new information. Weaker judges fall for it nearly every time, while GPT-4 falls for it under one time in ten.

The third is self-enhancement: the suspicion that judges prefer their own outputs. Evidence here is thin, with Claude showing the largest apparent self-love and GPT-3.5 showing none. The paper closes honestly: inconclusive, pending cleaner controlled experiments.

Calibration is where trust comes from. The baseline is votes from fifty-eight experts. GPT-4 agrees with the human majority eighty-five percent of the time. Human experts agree with each other eighty-one percent of the time. The judge ties humans rather than beating them — read that as a ceiling statement.

Auto-generating the rubric: AutoMetrics

Monday's third anchor changes the move: instead of hand-writing rubrics, generate the metrics. Ryan et al.'s AutoMetrics (note first author Michael Ryan teaches this course) starts from MetricBank's 48 ready-made metrics, generates LLM-judge criteria from lightweight human feedback, and composes the set against the human signal with regression. The test spans five tasks. Agreement with human ratings beats pure LLM-as-a-judge by up to a third. The price is under a hundred feedback points. The composed set doubles as a proxy reward matching verifiable rewards — prototype teams without money or traffic finally get a cheap, trustworthy optimization target.

Further reading: AutoLibra asks what the judge should measure

AutoLibra pushes the question one step earlier. Task success is blunt: an agent can finish while repeatedly ignoring constraints, or behave well until failing at the final step. AutoLibra grounds open-ended human feedback in concrete trajectory behavior, clusters similar positive and negative behavior into metrics with definitions and examples, then asks an LLM judge to score new trajectories +1, -1, or N/A. Coverage and redundancy select a compact set that captures what people cared about without repeating itself.

The metrics can become improvement targets. Across collaborative, social, web, and text-game environments, the best reported coverage varies from roughly sixty to nearly ninety percent; metric-guided prompt iteration and trajectory selection also improve success on specific benchmarks. Those results are not general guarantees. The study covers text-based observations and actions, most new labels come from the authors rather than diverse end users, and bias in the extraction model can harden into the rubric. MT-Bench and AutoMetrics ask whether a judge is reliable; AutoLibra asks whether users' actual concerns made it onto the exam.

Four guardrails: passing grading is not production safety

Wednesday's core evidence comes from PrivacyLens: acing the quiz still leaks in action. The theoretical footing is contextual integrity: privacy is not the secret itself but whether an information flow fits its context's norms. One sentence is harmless to a colleague and a disaster inside a letter to your manager.

Method-wise, the authors collect four hundred ninety-three privacy-sensitive seeds. Each seed is a five-tuple: data type, data subject, sender, recipient, and transmission principle. Seeds grow into vignettes, then into full tool trajectories inside a sandbox. Probing questions test the quiz layer while the final action tests the doing layer, scored separately.

The result is a cold splash. Even with privacy-enhancing prompts, GPT-4 leaks sensitive information in just over a quarter of cases. Llama-3-70B leaks in nearly four cases in ten. Answering well is not acting well, now with reproducible evidence.

Evals are the mock exam and guardrails are the exam proctors, with four standard pieces in practice. First, injection defense: treat tool outputs and retrieved content as untrusted input, echoing the MCP spec from Week 3. Second, red-teaming: grow many trajectories from one seed and actively hunt the leakiest path. Third, sandboxing: start every trial from a clean environment so experiments cannot wound the host. Fourth, permission: ask explicit consent before irreversible tool calls.

Attacks evolve: privacy offense and defense in simulation

Wednesday's other two anchors push the threat forward. Zhang and Yang (Diyi Yang again) search privacy risks through simulation: hostile agents coax sensitive information out over multi-turn dialogue, where dynamic conversation outruns pre-written defenses. Their setup lets offense and defense co-evolve inside simulation — an LLM optimizer rewrites instructions from trajectories, with parallel multi-thread search over the strategy space. Both sides evolve: attacks graduate from blunt requests to impersonation plus forged consent, defenses from plain rules to identity-verification state machines. The discovered plays generalize across scenarios and backbone models, ready to reuse for privacy-aware agents.

Deanonymization: off-the-shelf tools suffice

Li's deanonymization study stings more: in Anthropic's public Interviewer dataset, twenty-four scientist interviews mention published work. Six were linked back to specific paper authors. The attacker built nothing new — an LLM with search and agentic capabilities, a few prompts of cross-referencing. Safeguards break when decomposed into benign subtasks. Anthropic has been notified. Read it as the week's warning: once rich data is public, every anonymization assumption of the agent era needs recomputing.

Further reading: privacy guardrails need an alternative at each step

Contextualized Privacy Defense for LLM Agents does not treat privacy as keyword blocking. Whether the same datum may be shared depends on the recipient, purpose, and relationship. Static system prompts can fade during long executions, while a post-generation guard can reject an action without showing the agent how to share only the permissible part. CDI places a separate, potentially smaller instructor model after a tool result enters context. It reads the current situation, writes step-specific privacy guidance, then hands control back to the main agent. Training reuses frozen contexts around the first leak as RL environments, optimizing privacy preservation, helpfulness, and appropriate disclosure together.

Across 115 simulated configurations, the best setup reports 94.2% privacy preservation on unseen scenarios. A fresh attack round pushes that figure down to 79.5%, still above the paper's prompting and guarding baselines. This is not deployment proof: scenarios derive from PrivacyLens and model expansion, an LLM judge labels leakage, and the result depends on specific agent, instructor, and attacker models. The durable idea is that a guardrail need not only say yes or no at the end; it can shape a safer but still useful action while that action is forming.

Further reading: prompt injection has no silver bullet

OpenAI's Understanding Prompt Injections starts from a structural problem: third-party text in webpages, documents, and email enters the same context as trusted instructions. When an agent also holds private data, tools, and long-horizon autonomy, malicious text can cause exfiltration or unauthorized action rather than merely a wrong answer.

The official proposal is defense in depth: instruction hierarchy and safety training, rapidly updated monitoring, sandboxing and privilege separation, human confirmation for consequential actions, red teaming, and a bug bounty. Every layer can miss, so the goal is to minimize authority and blast radius rather than trust a stronger prompt. The article explicitly calls prompt injection an ongoing open problem; no single layer has solved it. A Week 8 judge can catch only part of the failure surface, and one malicious external string should not gain high-impact authority when the judge misses.

Further reading: what happens after an eval crosses the line

The assigned link is Anthropic's 2023 Responsible Scaling Policy v1. Borrowing from biosafety levels, it introduces AI Safety Levels: as models approach capabilities that could create catastrophic harm, security, deployment, evaluation, and red-team requirements must rise with them. The core governance commitment is not “one eval certifies safety.” Capability evaluations feed stop/go decisions, and scaling should pause when safeguards lag.

This is a voluntary, revisable company policy, and the assigned reading is historical. At this update, Anthropic's current policy page lists v3.4, effective July 2026; v1 details cannot be described as unchanged current commitments. Its Week 8 value is the governance interface: the first half of the week builds judges and metrics, while RSP asks who may deploy, which protections must rise, and when development must stop after a threshold is crossed.

What to do: ship one judge score and one permission check

What to do: give your own agent one judge eval plus one guardrail. For the eval, pull twenty real tasks from the bug tracker, check correctness deterministically, and judge tone and completeness with one natural-language assertion each, rerunning weekly to catch regressions. For the guardrail, start with exactly one rule: irreversible moves like sending mail or deleting files need explicit permission first. This week's deliverable is one live judge score plus one permission check that has blocked a real incident.

Where it sits in the course

Week 7 spreads out the benchmark map, and Week 8 hands over the tools: automated judges guard quality, guardrails guard safety. This mapping follows the official CS329Z schedule, with the paper video due Friday. Monday's three anchors are the eval guide, MT-Bench, and AutoMetrics. Wednesday's three are PrivacyLens, simulation offense-defense, and deanonymization. Once Friday's paper video lands, only Demo Day remains. The judge eval is the project's quality gate, and the guardrail is its on-stage insurance.

This week's course material

• Mon 11/9 LLM-as-Judge & Evaluation Infrastructure: anchor readings Demystifying Evals, MT-Bench, and AutoMetrics, plus Zhu et al., AutoLibra; all are covered above.
• Wed 11/11 Agent Safety & Guardrails: anchor readings PrivacyLens, Zhang & Yang, and Li, plus Wen et al., Contextualized Privacy Defense, OpenAI's prompt-injection threat model, and Anthropic's RSP v1; all are covered above.
• Course schedule: CS329Z site

Update log

• 2026-09-12: Added substantive guides to AutoLibra, CDI, prompt injection, and RSP, including the historical-version boundary for RSP.

References

• On this site: Week 7: evals and benchmarks, Week 3: MCP and DSPy, Stanford CS329Z course guide
• Course: CS329Z schedule
• Sources: Grace et al., Demystifying Evals for AI Agents, Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023, Ryan et al., AutoMetrics, Zhu et al., AutoLibra, Shao et al., PrivacyLens, NeurIPS 2024, Zhang & Yang, Searching for Privacy Risks via Simulation, Li, Agentic LLMs as Powerful Deanonymizers, Wen et al., Contextualized Privacy Defense, OpenAI, Understanding Prompt Injections, Anthropic, Responsible Scaling Policy v1, Anthropic, current RSP
• Tools: FastChat llm_judge, PrivacyLens