AI Agent Arxiv Digest — 2026-06-04

🌏 中文版

Today's Overview

Three papers tackling "how to build more reliable, evolvable Agent systems" from different angles: the first reveals real LLM call costs in multi-model Agent systems through execution traces, giving platform engineers hard numbers to work with; the second proposes treating the entire memory pipeline as self-evolving code to fix memory-architecture drift in long-running tasks; the third exposes evaluation blind spots—current Agent continual learning benchmarks can't tell whether agents actually learned anything—and introduces a more rigorous controlled stream framework.

Terms to Know Before Reading

Paper 1 | Characterization of Multi-Model Agentic AI Systems on General Tasks via Trace-Driven Simulation

Authors: MiroMind AI research team (the paper studies MiroThinker and OWL; full author list not fully available in search results)　·　arxiv: 2606.01725 Links: arxiv · alphaxiv

TL;DR

The first token-level trace dataset (GAIATrace) recording "how many LLM calls it actually took to complete each task" in real Agent systems, plus a low-cost replay simulator (Vidur-Agent).

Read Priority

Must-read. If you're building Agent platform infrastructure, this gives you the first reliable system-level cost numbers; if you're building Agent products, this shows you the real LLM call scale of current SOTA systems.

Background

Modern Agent systems are increasingly complex: a single task may involve a "main model" for planning, multiple "sub-models" for specific execution steps, and numerous tool calls. The problem is that token consumption, LLM call counts, and workload distribution across models have almost never been systematically recorded. Without numbers, engineers can't locate bottlenecks, and researchers can't reproducibly compare different architecture designs.

Mid-Level Walkthrough

Problem

You want to optimize costs for a multi-model Agent system, but you don't know how calls are split between the main model and sub-models. When a task fails, which component broke? Previously, no data could answer these questions: each evaluation was expensive (requiring a full run), and results were hard to reproduce (LLMs are inherently stochastic).

Method

The researchers ran MiroThinker and OWL—two SOTA agentic systems—through the entire GAIA benchmark (a general-task evaluation suite covering web search, code execution, multi-step reasoning, etc.) and recorded complete execution traces at every step, including full reasoning tokens, tool call logs, and each sub-model's inputs and outputs. The resulting dataset is called GAIATrace. To enable low-cost reproduction, they also developed Vidur-Agent—a simulator that can "replay" GAIATrace traces.

Why It Matters

Platform engineers can finally answer "how expensive is a multi-model architecture, really?" with real numbers. This data also lets researchers test new scheduling strategies or optimization ideas without actually running models, dramatically lowering the research barrier.

Deep Dive

• GAIATrace scale: MiroThinker completed 103 tasks using 1,491 main-model calls + 591 sub-model calls; OWL completed 165 tasks using 2,669 main-model calls + 2,737 sub-model calls ⚠️ (numbers from the paper, but both studied systems are the authors' own products—potential selection bias)
• OWL averages ~16 main-model calls + ~17 sub-model calls per task, the most concrete multi-model Agent cost baseline available today
• Vidur-Agent is an Agent extension of the existing LLM inference simulator Vidur, supporting trace replay on simulated hardware environments and comparison of different scheduling strategies
• GAIA benchmark is the most widely used general-task evaluation suite for agentic AI, proposed by Meta et al., with three difficulty levels
• Relation to mainstream frameworks: GAIATrace currently only contains trace formats from MiroThinker/OWL, with no coverage of LangGraph, AutoGen, CrewAI, etc., limiting generalizability
• Limitation: Only two systems' traces—hard to generalize across all architectures; full token breakdowns from proprietary models can't be externally observed
• Adoption barrier: Whether GAIATrace will be publicly released and Vidur-Agent's compatibility with other agent frameworks remain to be confirmed

Reviewer's One-Line Take

The contribution is real and scarce—this kind of token-level system trace data is genuinely a gap—but both studied systems were developed by the authors themselves; wait for community replication on other architectures before treating the numbers as universal. Use them as order-of-magnitude references, not general-purpose standards.

Your Take-Away

• OWL's cost numbers (165 tasks completed → ~33 LLM calls per task on average) are currently the most citable concrete baseline when estimating multi-model Agent compute costs for stakeholders
• When designing observability features for agent systems, GAIATrace's trace schema defines which fields are most valuable for system analysis—a useful reference for log schema design

Paper 2 | MemPro: Agentic Memory Systems as Evolvable Programs

Authors: Qingshan Liu, Guoqing Wang, Wen Wu, Jingqi Huang, Xinqi Tao, Dejia Song, Jie Zhou, Liang He　·　arxiv: 2606.00619 Links: arxiv · alphaxiv

TL;DR

Treat an Agent's entire memory system—including the logic and code for "how to store" and "how to retrieve"—as a self-iterating program, letting the Agent learn from failures and update the memory architecture itself, not just the memory contents.

Read Priority

Must-read. Agent memory is the core bottleneck for long-running tasks. This paper directly addresses the fundamental pain point of fixed architectures and proposes a mechanistically novel solution. Engineers working on agent memory or RAG architecture design should read carefully.

Background

Long-running Agents need memory systems to record what they've done, their current state, and reusable knowledge—avoiding reasoning from scratch every time. The mainstream approach is the MCR Pipeline: fixed logic handles "writing to memory," and another fixed set of logic handles "reading from memory." The problem is that task types vary and failure modes are diverse; a fixed pipeline quickly becomes outdated—especially as the memory store grows over time, when original indexing and query strategies may become completely inadequate.

Mid-Level Walkthrough

Problem

Your Agent frequently fails in long-running tasks because it "can't retrieve the right historical memories," and you have to manually fix the memory logic every time. As tasks become more diverse and the memory store grows, this maintenance cost keeps climbing, and the originally designed pipeline may be completely unable to handle new failure modes.

Method

MemPro treats the entire MCR pipeline as "a runnable program" rather than a fixed prompt combination. The system maintains a version tree where each node is a complete memory system implementation, containing executable code and prompts. An "Evolving Agent" continuously diagnoses failure modes, generates targeted improvements for frequently failing areas, and validates which version performs better through actual execution. The entire process iterates continuously.

Why It Matters

This puts "the memory architecture itself" into a self-improvement loop, rather than just updating memory contents. As Agents face increasingly diverse tasks, having the memory architecture auto-adapt is critical for long-term stable operation.

Deep Dive

• Version tree design: Each node is a complete MCR pipeline implementation (code + prompts); the Evolving Agent selects the most promising node, diagnoses common failures, and generates improved child nodes
• Evolution targets more than just prompts: MemPro also modifies executable code (e.g., memory store indexing strategies, filtering logic), providing a much larger adaptation space than pure prompt-tuning
• Failure-mode guided editing: Changes aren't random but targeted based on diagnosed failure modes, making evolution more efficient than random search
• Relation to MemEvolve (2512.18746): MemEvolve explores meta-evolution; MemPro goes further on the dimension of pipeline-as-program
• Limitation: Version tree management itself has computational costs; the Evolving Agent is also an LLM, so diagnostic quality is limited by the underlying model's capability; no direct quantitative comparison with mainstream memory systems like MemGPT on public benchmarks ⚠️
• Relation to LangGraph/AutoGen: MemPro's evolution mechanism could theoretically serve as a plug-in memory layer, but integration effort and API compatibility are unclear
• Adoption barrier: Requires maintaining version tree infrastructure and designing automated test suites to evaluate version quality

Reviewer's One-Line Take

The concept directly addresses a real pain point with genuine architectural novelty, but lacks quantitative comparison with known baselines (MemGPT, typical RAG approaches) on public benchmarks. Currently reads more like a compelling architecture proposal; actual effectiveness needs follow-up replication.

Your Take-Away

• If your Agent frequently fails in long-running tasks due to memory retrieval failures, start by classifying failure modes ("remembered wrong," "wrong format," or "can't find it at all")—MemPro's diagnostic approach shows that targeted fixes to memory logic based on specific failure types are more efficient than a full redesign
• When designing memory modules, versioning "the memory access logic itself" (not just memory contents) is an architecture decision worth evaluating, especially for systems that need long-term evolution

Paper 3 | AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents

Authors: Yiheng Shu, Bernal Jiménez Gutiérrez, Saisri Padmaja Jonnalagedda, Yuguang Yao, Huan Sun, Yu Su (Ohio State University · Johns Hopkins University · Intuit AI Research)　·　arxiv: 2606.02461 Links: arxiv · alphaxiv

TL;DR

Most existing Agent continual learning benchmarks are too lenient—they can't distinguish "the Agent genuinely applied knowledge from earlier tasks to later ones" from "the later tasks were just easier." AgentCL fixes this by deliberately designing cross-task reusability.

Read Priority

📖 Skim. If you're evaluating or designing Agent continual learning and memory capabilities, this provides a more rigorous methodology. If you just want the gist, the TL;DR and deep dive points are sufficient.

Background

"Continual learning" for Agents means: after completing task A, the Agent can store solutions, discovered facts, or workflows so that task B can be completed faster and more accurately—instead of reasoning from scratch each time. This differs from RAG (querying an external knowledge base) in that it emphasizes "learning from one's own past action experience." The problem is that current evaluation task streams are designed too casually to confirm whether Agents are actually reusing knowledge across tasks.

Mid-Level Walkthrough

Problem

Existing benchmarks give Agents a sequence of tasks, but the inter-task relationships aren't deliberately designed (naive stream). Even if the Agent performs well on later tasks, you can't tell whether it "actually used what it learned earlier" or "that task was just inherently easier." The result: different memory designs show minimal performance gaps on these benchmarks, making it impossible to distinguish quality.

Method

AgentCL introduces "controlled task streams": deliberately designed so that sub-solutions, discovered facts, or workflows from earlier tasks can be reused in later ones. By comparing "controlled stream" vs. "naive stream," researchers can clearly measure which memory design actually works. Evaluation covers three domains: coding, deep research, and language understanding/reasoning. The core metric is transfer gain: how much does the success rate on later tasks improve when earlier task memories are available?

Why It Matters

This is an upgrade to Agent evaluation methodology. With AgentCL, you can more reliably determine "this memory module design genuinely makes the Agent improve over time," instead of spinning in noise. For platform developers, this directly affects which memory architecture you should adopt.

Deep Dive

• Transfer gain as core metric: Measures not just task completion rate, but "how much did the success rate on later tasks improve with memories from earlier tasks"—the only way to truly gauge memory effectiveness
• Controlled stream design principles: Earlier tasks produce reusable sub-solutions, evidence, or workflows; later tasks are deliberately designed to require these for efficient completion, ensuring "reusability" is a controlled variable
• Key finding—naive vs. controlled stream: The paper found that naive streams have limited discriminative power across memory designs; controlled streams can more clearly differentiate a design's "plasticity"—i.e., the memory architecture's ability to genuinely learn from past tasks
• Three evaluation domains: coding (programming problem streams), deep research (multi-step research tasks), language understanding/reasoning
• Relation to Paper 2 (MemPro): AgentCL's framework could be directly used to evaluate the effectiveness of self-evolving memory systems like MemPro—the two papers complement each other
• Limitation: Controlled streams require manual curation, making scaling expensive; evaluation is primarily English-language; full result numbers are not yet fully disclosed in searchable sources ⚠️
• Implications for LangGraph/AutoGen developers: The design principles here can serve directly as methodology for "Agent memory module A/B testing"—use controlled streams to verify whether a memory upgrade genuinely produces transfer gain

Reviewer's One-Line Take

A solid methodological contribution that fills a real gap in existing evaluation; however, controlled stream design leans toward academic experimental settings—whether it can quickly extend to real task distributions in production, and multi-language/multi-domain applicability, remain to be validated.

Your Take-Away

• Next time you compare two Agent memory approaches, design your test tasks so that "solutions from earlier tasks can be reused in later ones" (controlled stream principle)—this way you're actually testing memory effectiveness, not just task difficulty variance
• Transfer gain tells you more than raw task completion rate about whether a memory module is truly delivering improvement—worth adding to your evaluation metric set

References

• arxiv:2606.01725
• arxiv:2606.00619
• arxiv:2512.18746
• arxiv:2606.02461