Agent sandboxes, credentials, and network boundaries
Probe OS-enforced file and network denials separately from scoped, expiring authorization. Application allowlists are not OS sandboxes, and transport choices do not restrict process capabilities.
ai
1239 postsHow to verify defenses: red teaming, regression, and monitoring
Replay an independent authorization matrix against real HTTP responses before and after a fix, correlate audit events, and report model behavior, enforcement, and production monitoring separately.
An available tool does not authorize this invocation
Bind a ticket export to its caller, resource, action, destination, and approval. Test substituted arguments, revocation, expiry, and replay.
AI Says It Found a Vulnerability. What Evidence Do You Need?
Treat AI suggestions as hypotheses. Preserve actual inputs and outputs, check the target data against the authorization policy, and use normal and patched controls to rule out false positives.
Let AI look for vulnerabilities: from clues to testable hypotheses
Turn AI suspicions into falsifiable authorization hypotheses with same-tenant and cross-tenant controls.
How Can You Demonstrate That AI Helps Security Work?
Freeze independent labels before measuring false positives, false negatives, abstention, human effort, and unsafe actions. Faster closure does not establish correct closure, and rule replay is not model evidence.
Seeing What an Agent Does: Drawing the Execution Trace from stream-json | Harvard CS2680 Assignment 1
CS2680 Assignment 1 has you build a web page that drives Claude Code: submit a task, watch live what the agent reads, what it runs, and how it turns out, then show how much it cost, how many seconds, and how many turns. The key is reading the event stream from `claude -p --output-format stream-json --verbose`: tool_use and tool_result pair up by the same id, and subagent events hang back under the step that dispatched them via parent_tool_use_id. This post breaks down the seven requirements, shows 40 lines of Python that build the events into a tree and tell success from failure, then covers submission, grading, and the places development most often gets stuck.
Building a Coding Agent Harness by Hand: Four Tools and a Loop | Harvard CS2680 Assignment 2
CS2680 Assignment 2 (Mads-Loop) has you write your own coding agent harness: with a fixed small model, qwen3.6-35b-a3b, it fixes real SWE-bench Pro issues inside Docker containers and logs every step in a required format. The assignment page says plainly not to trust the design it hands you, and some of the 10 tasks can't be solved. This post covers the spec, the places on the page worth doubting, the five places a harness most often breaks and how to handle each (with a snippet that has actually been run), how to build your own four-task eval set, and how to tell whether a task really is unsolvable. No official solution.
Saving Money and Time Without Losing Tasks: A Cost-Optimization Competition | Harvard CS2680 Assignment 3
CS2680 Assignment 3 (Mads-Opt) is a harness competition: on the same SWE-bench Pro tasks, who solves the most, spends the least per solved task, and waits the least per solved task. The three metrics aren't weighted into one score. Instead submissions are ranked in skyline (Pareto) layers: if nobody is at least as good as you on all three and better on at least one, you're rank 1. This post covers the container setup and hard limits, how ranking and scoring work (with a worked example), a first estimate of what one task costs using the model prices on the assignment page, and how Lecture 8's cost and speed levers map onto the assignment. No official solution and no specific strategies.
Modern AI Systems, From Using Agents to Optimizing Them | Harvard CS2680 Course Overview
Harvard CS2680 follows one coding agent and takes it apart from the top down. You start by watching Claude Code run, then write your own loop, pick tools and manage context, grow it into multiple agents, add memory, learn to evaluate it, and finally cut its cost and latency. In the second half of the term you go further down and serve the model yourself. This post covers what Part I's eight lectures and three assignments teach, works one example by hand to show why every extra agent step costs more, and ends with a prerequisite check and a reading order.
Agents Are a Systems Problem: From One Inference Call to a Chain of Tool Calls | Harvard CS2680 Lecture 1
CS2680 Lecture 1 (2026-09-02) starts with how AI systems differ from traditional systems: computation is split across CPUs and GPUs, floating-point math doesn't give the same answer every time, and state is large enough that it has to move between GPUs. It then compares an agent to a new kind of computer (the LLM is the CPU, the context window is memory, the harness is the operating system) and shows that one agent request becomes a long chain of inference → tool → inference. That makes tokens per second the wrong target; what matters is task success rate, end-to-end latency, and cost per success. This post works through two agents' numbers to show how those metrics are used.
How Claude Code Works: The Loop, Memory, Tools, and Extensions | Harvard CS2680 Lecture 2
CS2680 Lecture 2 (2026-09-09) takes Claude Code apart. You type one request and it calls the model many times. Each time, the model only proposes a tool to run; the harness checks permissions, runs it, and puts the result back into context, and when the context fills up it gets compacted. The second half covers memory (the CLAUDE.md you write and the auto memory Claude keeps) and four extensions: skills cover how to do things, MCP reaches outside systems, hooks cover what must always happen, and plugins handle packaging. This post first shows what one API call looks like, then walks the whole loop through fixing one test.
Running Many Agents at Once: Subagents, Agent Teams, and Programming 3.0 | Harvard CS2680 Lecture 3
The first half of CS2680 Lecture 3 (2026-09-14) is about scaling yourself up. It uses two plugins, Superpowers and Caveman, to show how skills, hooks, and a proxy fit together, then covers three directions to scale: run agents in parallel to buy wall-clock time, delegate to subagents to buy focus, and let one agent run long to buy scope. The second half is about the cost of scaling: when you have an agent write code from natural language, uncertainty seeps in from four layers (meaning, context, model sampling, the tool environment), and the remedies are to define clearly, separate proposal from execution, and tighten verification. This post works through "three bugs in three worktrees" to show what parallelism actually saves.
Designing Your Own Agent: Tool Design and Context Engineering | Harvard CS2680 Lecture 4
CS2680 Lecture 4 (2026-09-16) switches from using agents to designing them. An agent is a model plus a loop, and you decide which tools it gets and what it sees at each step. The tools half covers four ways to find code, four ways to edit it, the power and risk of a shell, and a set of design principles: few orthogonal tools, purpose-built over generic, token-cheap output, errors that teach. The context half covers why context is a finite resource, four ways to manage it (from guarding the door to summarizing as a last resort), and five ways context fails. It closes with a paper showing that when context is tight, the same model with only a different harness can solve 70% to more than twice as many tasks.
Multi-Agent Architecture: Topology, Task Decomposition, and Communication Between Agents | Harvard CS2680 Lecture 5
CS2680 Lecture 5 (2026-09-23) is about designing multi-agent systems. There are four reasons to split work across agents: specialization, isolation, parallelism, and context management. Any architecture can be described by three questions: who coordinates, how the work is split, and how messages move. The lecture lays out eight architectures, then covers six ways to decompose a task, a granularity formula for deciding whether to split, six common decomposition failures, and why communication is really a distributed systems problem: messages get lost or resent, and two agents end up changing the same thing at once. It closes by contrasting A2A with MCP, with "Is multi-agent really necessary?" as the first design principle.
Agent Memory and Reliability: What to Write, How Long to Keep It, and Which Layer Failed | Harvard CS2680 Lecture 6
The first half of CS2680 Lecture 6 (2026-09-28) is about agent memory. Keeping one fact means deciding, in order, whether to write it, what form it takes, where it lives, when to retrieve it, whether to generalize from it, and whether it still holds, plus who owns the memory. Don't simply delete outdated facts; record when they were true and where they came from. The second half is about reliability: at 95% success per step, a 20-step run is fully correct only about 36% of the time. Failures can be traced to five layers (task spec, harness, model, tool, environment), each with its own fixes. The two halves meet at one harness-layer failure: treating a stale note as fact.
Evaluating Agents: Benchmarks, pass@k, and Data Leakage | Harvard CS2680 Lecture 7
CS2680 Lecture 7 (2026-09-30) is about measuring whether an agent is any good. First, be clear whether you are measuring a component or the whole task, and whether "success" means a correct answer, working code, or an environment left in the right state. On metrics, pass@k asks whether at least one of k tries succeeds, while pass^k asks whether all k succeed; from the same set of trials the two can come out as 0.9 and 0.3. Then come eight representative benchmarks and what each one measures, and finally when a score can't be trusted: the answer may be in the training data, the agent may look it up online, it may be sitting in the environment's git history, and the grader itself can be gamed.
Making Agents Cheap and Fast: Cost Breakdown, the Critical Path, and Harness Architecture | Harvard CS2680 Lecture 8
CS2680 Lecture 8 (2026-10-07) has two halves. The first takes apart an agent's bill: cost is a product of users × sessions per user × turns per session × steps per turn × tokens per step × price per token, and the last four are the ones you can cut. The students' own Claude Code logs show 97.6% of tokens are cached re-reads, so keeping the prefix stable matters more than keeping the prompt short. Speed is set by the critical path, not the sum of every step. The second half compares three harness architectures (monolithic, modular, microkernel) and the design choices of eight open-source and commercial harnesses. This post works one bill by hand: put a timestamp at the start of the system prompt and the same task goes from 23.6 units back to 65 or more.
11 Days with /autocompact 400k in Claude Code: 16–22% Less Usage, No More Rework
On 1M-context models, Claude Code waits until about 967K tokens to auto-compact, so late messages each carry nearly a million tokens of history. I set /autocompact 400k as Lydia Hallie suggested and used it for 11 days: roughly 16–22% less usage, and no noticeable increase in rework or corrections from lost details. This is one person's observation on one kind of work, not a general finding.
Trying vLLM on a Free Google Colab T4: See What an Inference Engine Does in 15 Minutes
A Mac has no NVIDIA GPU, so it can't really run vLLM — but Colab's free T4 (compute capability 7.5, exactly vLLM's minimum) can. Serve the 0.6B Qwen3 as an OpenAI-compatible API with vllm serve, fire 8 requests at once, read /metrics, and you can watch continuous batching happen.