🌏 中文版
Video status: Videos included. Lecture text is available; caption type, listening checks, and the shadowing player still require verification.
An agent runs for two hours and produces a polished report. Has it completed the work? Stanford CS329A, Autumn 2025 Lecture 17, evaluates complete tasks: sustained execution, usable artifacts, and verifiable evidence.
This guide uses the official readings, Measuring AI Ability to Complete Long Tasks, GDPval, and DeepScholar-Bench, alongside the lecture's text transcript. It is for readers building tool-using agents who need concrete acceptance criteria.
For prerequisite context, first read search and deep research.
Whose time does the horizon measure?
The first paper, from METR, uses human completion time as a task-difficulty scale and estimates model performance at a specified success rate. A 50% time horizon is the human task duration corresponding to a predicted one-half success probability. It is not the maximum time the agent can remain running or a guaranteed duration of autonomous work.
Human duration enables comparisons, but tool familiarity and task details also affect difficulty. A deployed service may require substantially higher reliability, so a 50% horizon alone cannot establish service quality.
Why long workflows fail
Here is an original simplified example. A workflow has ten necessary steps, each independently successful with probability 0.95, without retries. Complete success is 0.95 raised to the tenth power, approximately 0.599. Averaging ten 95% rates does not give the probability that every step succeeds.
Allow one independent retry per step. Both attempts fail with probability 0.05 × 0.05 = 0.0025, giving per-step success 0.9975. Complete success becomes approximately 0.975. This illustrates recovery's value, not an empirical agent reliability estimate. Missing data and incorrect specifications usually do not disappear through identical independent retries.
Professional artifacts and research need different measures
GDPval uses professional tasks and artifacts, primarily comparing model outputs against human baselines through expert judgments. It asks whether a document, spreadsheet, or design is usable. Artifact comparisons do not establish that an entire occupation can be replaced; jobs also involve interaction, changing requirements, and responsibility.
DeepScholar-Bench separates knowledge synthesis, retrieval quality, and verifiability. A well-formatted citation can name a nonexistent paper, or a real paper that does not support the claim. Separate finding a source from producing a supported argument to diagnose retrieval and synthesis failures.
Worked example: accepting a supplier comparison
This is an original task. Inputs are three supplier documents. The deliverable is a comparison table and recommendation covering four necessary facts: A's monthly price, B's delivery time, C's support period, and whether tax is included. Preserve sources and mark unknown information explicitly.
One report covers the first three facts but invents tax inclusion. Only four of its six major claims are supported by cited material. Simple coverage is 3/4=75%; claim support is 4/6, approximately 67%. These are teaching criteria, not the papers' formal scores. Perfect formatting does not satisfy an acceptance rule requiring evidence for every decision-critical claim.
Do not simply choose a judge that prefers the report. Check whether the inputs contain tax information. If absent, change the claim to unknown. If present, inspect whether document reading omitted it. Evaluation should point to a concrete repair.
Exercises and answers
With twenty independent steps at 0.95 success each and no retries, complete success is approximately 0.358. Failures accumulate. If errors are correlated, the independence model is unsuitable; estimate performance through repeated actual runs.
If all four required facts appear and all six citations refer to real sources, does the report pass? Not necessarily. Check that citations support claims, dates and units match, and the recommendation addresses the original request. Source existence is not claim support.
Write an acceptance card today
Choose a real agent task. Record inputs, deliverable files, required information, unacceptable errors, and the desired success rate. Preserve tool logs and artifacts across repeated runs. Distinguish reading failures, reasoning errors, and missing deliverables, then choose repairs from those categories rather than report only a mean score.
English practice: audio shadowing pending
The official video's text transcript is available for content comparison. Caption type, timestamps, listening checks, and three replayable segments remain unverified. Shadowing is pending. The calculations and supplier task are original examples, not lecturer experiment results.
Lecture video source
References
Loading...