Reading Stanford CS329Z Week 11 (Finale): From Waiting for Orders to Acting First — Proactive Agents and Demo Day

🌏 中文版

Picture two assistants. The first waits for your orders and never moves until told. The second watches: you receive a wedding invitation from a friend, and it already has suit-rental options and a budget waiting for you. The first is reactive, the second is a proactive agent, and Week 11 Monday is about the second kind.

The difference is not model size but who moves first. A reactive agent's loop starts with your instruction; a proactive agent's loop starts with its observations of you. Observation takes a pair of eyes, inference takes a user model, and acting takes rules for when to interrupt. Those three are exactly the three protagonists of this week's main reading.

Week 11 is arranged like a closing act. Monday (Proactive Agents) assigns Shaikh et al.'s General User Models, with a follow-up paper on Next Action Prediction plus privacy-and-trust discussion. Wednesday (Open Problems & Final Demos) has no single assigned reading: the course lays out multimodal agents, web and computer use, science agents, long-running architectures, and production observability, converging on three open directions. My Wednesday write-up below is framed as the course's view, not any one paper's claim. Paper-video peer reviews are also due Monday, and Demo Day lands in finals week under the theme Making Life at Stanford Better with Agents.

GUM: turning screen traces into a user model

GUM takes any unstructured observation, such as screenshots, as input. Its output is a set of natural-language propositions, each carrying a confidence score. Seeing a wedding invite, it writes down that the user is invited to a friend's wedding; seeing repeated draft edits followed by related-work reading, it writes down that the user is stuck on collaborator feedback. High confidence for the former, low for the latter, all labeled honestly.

The architecture is four modules. Propose turns observations into propositions, Retrieve pulls related old propositions in as context, Revise updates them as new evidence arrives, and Audit blocks anything the user would not want recorded. Screen understanding runs on Qwen 2.5 VL, proposition reasoning on Llama 3.3 70B, both on the open route. The paper deliberately picks open models so data never has to travel to a third-party platform. In the implementation Llama runs on private servers, and only the user can access the propositions.

The examples persuade. The first author's GUM found him a suit-rental spot for a wedding, automatically constrained by his budget. GUM can also complete a prompt's missing context — which section you are writing, which papers you have read — so a bare "help me with this section" just works. That is the same grounding Week 2's RAG chases, only the source changes from a retrieval index to traces of your computer use.

One set of numbers is worth remembering. In an email-based evaluation, propositions averaged about 76% accuracy, and the most confident ones were all correct. A separate multi-day screen deployment replicated the calibration, and some participants asked to keep running the system afterward.

Gumbo: the assistant that acts before asking

Gumbo is the demo application built on GUM. It watches the screen, grows a GUM, and mines it for suggestions. Before interrupting you, it computes an expected utility: the probability the suggestion helps times its benefit, weighed against the cost of a wrong interruption. The math comes from Horvitz's mixed-initiative framework, and GUM supplies the off-the-shelf cost-benefit estimates it always lacked.

To stop suggestion floods, a rate limiter caps interruptions at one per minute. Suggestions that pass the bar get executed as far as possible — searching, running code, organizing files — before anything is shown. Thumbs up or down feed straight back into the GUM as new observations. The paper is honest about boundaries too: computer use stayed disabled during evaluation because it was too slow and buggy.

Next Action Prediction on the schedule points at something concrete. A follow-up by Shaikh et al. formalizes predicting the next action from multimodal computer-use traces — exactly the capability Gumbo's pipeline points at. See the Course Material box at the end for the full mini-review.

Privacy and trust: stronger powers, costlier mistakes

The Audit module filters observations by contextual integrity. Plainly put: would recording this information fit the setting it appeared in? The evaluation's verdict is honest: mostly compliant, but violations are bad when they happen. That is why Week 11 pairs privacy and trust with proactivity. When a reactive agent errs, you are at least present; when a proactive agent errs, you are often away.

The paper gives the privacy paradox its own section. The same participants wanted an assistant that remembers enough, yet flinched at seeing their habits written out as propositions. There is no technical fix here, only a tradeoff: the more it remembers, the more it helps — and the more it can get wrong.

This also echoes Week 1's systems view. None of GUM's four modules is the model itself; all are engineering around it: observing, auditing, retrieving, revising. Good results are still stacked up by systems — except this time the system is about you.

Wednesday: three open directions and production reality

Restating the caveat: this section is the course's synthesis, since Wednesday assigns no main reading. The course converges the frontier on reliability, scalability, and interpretability. Mapping earlier topics onto them, roughly: how long-running agents avoid drift and deadlock, what happens when web and computer use scale up, and whether behavior can be traced and explained. Scale has concrete yardsticks. OSWorld moves real computer tasks into a reproducible OS environment and scores execution results instead of text matches. The best model at the time succeeded on only about 12% of tasks, bottlenecked on GUI grounding and operational knowledge. On the web side, WebShop tests language grounding with crowd-sourced shopping instructions in a simulated store, where the best model succeeded on under 30% — even "buying the right thing" is still shaky.

Production observability is Wednesday's other half. Tracing, monitoring, and cost management form the trio; HW2 trained half of it through evaluation, and the rest must be in place before Demo Day. Research questions decide how far a project can go; observability decides whether the live demo survives. To push a Gumbo-style prototype toward an operable system, compare open-source OpenClaw: a personal assistant running on a local Gateway, with models and chat channels as swappable plugins.

What to do: the Demo Day checklist

What to do: turn tracing fully on and keep one replayable trace per rehearsal. Watch three things in monitoring: error rate, latency, and token spend. Put a hard cap on cost with automatic cutoff past the limit. Keep high-risk tools off by default and enable them by hand only for the live demo. Finally, prepare one failure case — a failure mode from your midway report, told well, persuades better than successes alone.

Series retrospective: 11 posts and one recess week

1. Course guide: build from scratch before frameworks, both homeworks and the syllabus git history in full.
2. Week 1: stop tuning only the model; good scores are stacked up by compound systems.
3. Week 2: separate workflows from agents first, then hand-build a first RAG.
4. Week 3: MCP standardizes the tool plug, DSPy turns prompts into compilable programs.
5. Week 4: ReAct fixes the loop as think–act–observe, while MemGPT turns memory into OS-style layered structure.
6. Week 5: multi-agent collaboration and the three optimization axes — prompts, weights, or inference compute.
7. Week 6: the data flywheel spins up as HW2 drops and HW1 comes due.
8. Week 7: data selection and benchmark design, with evaluation converging on the 4-tuple.
9. Week 8: LLM-as-judge and safety guardrails, where the graders themselves get graded.
10. Week 9: interface is performance — SWE-agent proves editor design decides scores, while OpenHands turns sandbox and evaluation into a shared base.
11. Week 10: Thanksgiving Recess; both meetings are canceled with no required or additional readings, so the series does not manufacture an empty guide.
12. Week 11 (this post): from waiting for orders to acting first, closing with proactive agents and open problems.

Course Material box

• Monday 11/30 Proactive Agents: main reading GUM (covered above); further reading Shaikh et al., Learning Next Action Predictors from Human-Computer Interaction. It formalizes predicting the next action from multimodal computer-use traces and introduces LongNAP, a model combining parametric and in-context learning. The data comes from continuous phone use by 20 users. Vision-language models labeled over 360K actions. An LLM-as-judge scores predicted versus actual next actions for similarity, and LongNAP clearly beats supervised-finetuning and prompted baselines.
• Wednesday 12/2 Open Problems & Final Demos: no single main reading; further reading OSWorld (open-ended computer tasks in real operating systems) and WebShop (language grounding in a simulated store) — concrete numbers in the Wednesday section above.
• Week 10 (Nov 23 and Nov 25): the official schedule marks both meetings No Class — Thanksgiving Recess, with empty required and additional reading lists.
• Schedule source: CS329Z schedule, Week 11

Update log

• 2026-09-12: Made the Week 10 Thanksgiving recess explicit in the series map and course-material checklist so the Week 9-to-11 jump no longer looks like a missing post.

References

• On this site: Stanford CS329Z course guide, Week 1: stop tuning only the model, Week 2: workflows versus agents, Week 3: plug tools in, swap frameworks up
• Course: CS329Z schedule
• Sources: Shaikh et al., Creating General User Models from Computer Use, UIST 2025, Shaikh et al., Learning Next Action Predictors from Human-Computer Interaction, GUM project page and open-source package, Horvitz, Principles of Mixed-Initiative User Interfaces, CHI 1999
• Venue: UIST 2025