AI Agent Arxiv Digest — 2026-08-08

🌏 中文版

Today's Overview

Today's three papers dissect the same question from different angles: how exactly do things go wrong when agents learn from past experience? Memory Reward Inflation shows that memory rewards self-inflate — agents assign high scores to incorrect experiences, then preferentially reuse them, creating a positive feedback loop. RoMeRL approaches from the engineering side of memory systems, finding that as interaction history grows, limited feedback gets diluted across an ever-expanding state space, causing irrelevant memories to be incorrectly reinforced. ToolLIFT shifts focus to tool planning, showing that building graphs directly from tool-level trajectories locks planning into specific tool sets, preventing transfer. The combined message is clear: agent self-improvement can't just be "save it and reuse it next time" — memory rewards need deflation, state spaces need compression, and tool experience needs abstraction.

Terms to Know Before Reading

Paper 1 | Memory Reward Inflation: Hidden Degradation in Self-Improving Agents

Memory Reward Inflation in Self-Improving LLM Agents

Mohammad Asadolahi, Amir Amini, Samira Talebi et al. (University of Isfahan)　·　arxiv: 2608.00017

Links: arxiv · alphaxiv

TL;DR

When self-improving agents use LLMs to score their memories, an "Echo Gap" emerges — incorrect experiences receive inflated rewards, get preferentially reused, and create a positive feedback loop. The label-free LUCID algorithm improves execution accuracy from 54.0% to 56.9% on BIRD.

Read Priority

Must-read — if your agent has any form of experience memory with self-evaluation, this paper directly exposes a pitfall you may already be stepping into.

Background

An increasing number of agent frameworks (Voyager, MemGPT, Reflexion) have agents store successful past experiences and retrieve them for similar tasks. The problem: in deployment environments without ground-truth labels, rewards can only come from LLM self-evaluation. Prior work assumed this self-evaluation was roughly reliable, but no one had systematically tested that assumption.

Intermediate Guide

• Problem: Imagine a student grading their own exam. They got a question wrong but felt confident, so they gave themselves a high score. Next time they encounter a similar question, they reference this "high-scoring" answer — and make the same mistake again, giving themselves another high score. This is the Echo Gap.
• Method: The authors treat scores in the memory store as "proxy rewards" and prove that when the scorer's errors correlate with the original self-evaluation bias (i.e., violating the "Error Independence Assumption" or EIA), inflation cannot be corrected. The LUCID algorithm reduces inflation through decorrelation techniques without requiring labels.
• Why it matters: This isn't a theoretical concern — any system using LLM self-evaluation to manage memory may be accumulating this bias right now. And it doesn't naturally decay; it only snowballs.

Key Details

• BIRD text-to-SQL benchmark: LUCID reaches 56.9%, self-evaluated memory agent 54.0%, memoryless agent 52.4%
• Inflation occurs under similarity-based retrieval too (not just score-based ranking) ⚠️ (author-reported, awaiting external replication)
• Echo Gap persists across model families — not a model-specific issue
• Low barrier to adoption: LUCID is a post-processing algorithm that can be layered onto existing memory systems
• Directly relevant to LangGraph / MemGPT memory modules — any memory-augmented agent should check for this
• Limitation: validated only on text-to-SQL so far; effectiveness on more complex multi-step tasks remains untested

Reviewer's One-Liner

Problem definition is clear and the formalization is solid — the Echo Gap and EIA theoretical framework is convincing. But BIRD is a relatively structured task, and whether the +2.9 point gain holds on open-ended agent tasks is the key thing to watch.

Your Take-Away

• If your agent has experience memory + LLM self-evaluation: immediately check the actual accuracy of high-scoring memories. If accuracy declines over time, that's a sign of inflation
• If you're designing a memory evaluation pipeline: the scorer must be decorrelated from the executor's bias — use a different model, different prompt, or external validators

Paper 2 | RoMeRL: Compressing Agent Memory State Space with Fixed-Dimension Coordinates

RoMeRL: Balancing Feedback Coverage and the Memory-Reward Trap in Self-Evolving Agent Memory via Reduced-Order Utility States

Yi Yang, Zhennan Chen, Yihong Zhuang et al.　·　arxiv: 2608.02508

Links: arxiv · alphaxiv

TL;DR

Replaces infinitely growing trajectory indices with fixed-dimension semantic coordinates, compressing each memory's utility into a low-dimensional space of "success/failure × memory dynamics" — Cold-Q ratio drops 80%, memory size shrinks 84.4%, LLM calls decrease 21.1%.

Read Priority

Must-read — directly addresses the problem raised in Paper 1 with an engineering-ready solution.

Background

The mainstream approach for agent memory systems: store one trajectory per interaction, update its utility value via reinforcement learning. The problem is trajectory count grows linearly with time, and limited feedback gets spread across ever more states, leaving most memories never effectively updated — this is the "memory-reward trap."

Intermediate Guide

• Problem: Imagine a librarian who receives a few new books every day but only gets one sticky note per day to mark good or bad. As books accumulate but sticky notes stay constant, most books end up unrated — you don't know if they're good, but they still get recommended to readers.
• Method: RoMeRL no longer maintains independent utility values for each trajectory. Instead, it defines a fixed set of "semantic coordinates," each representing a memory state under a task type (split into positive/negative × dynamic features). New experiences don't add coordinates; they update or replace existing ones.
• Why it matters: This transforms memory management from "infinite accumulation" to "bounded maintenance" — feedback density increases 6x, meaning each memory unit receives significantly stronger learning signal.

Key Details

• ALFWorld + LifelongAgentBench: highest overall average score
• Cold-Q ratio drops 80% (drastically fewer memories that were never effectively updated)
• Feedback density increases ~6.0x
• Maintained memory size reduced by 84.4% ⚠️ (author-reported, awaiting external replication)
• LLM calls reduced by 21.1% — leaner memory means more efficient retrieval
• Medium barrier to adoption: requires defining semantic coordinate dimensions and update rules
• Compatible with Reflexion and MemGPT memory modules
• Limitation: semantic coordinate dimensions need to be set manually; automatically determining optimal dimensions is an open problem

Reviewer's One-Liner

Solid theory and experiments — "fixed-dimension compression" is an elegant engineering solution. The open question is whether semantic coordinate dimensions can automatically adapt across different task types.

Your Take-Away

• If your agent's memory keeps growing but performance doesn't improve proportionally: RoMeRL's "fixed coordinates + replacement updates" architecture is worth adopting directly
• If you're evaluating memory system quality: Cold-Q ratio is an excellent diagnostic metric — above 50% means your memory system is spinning its wheels

Paper 3 | ToolLIFT: Abstracting Tool Experience into Transferable Functional Workflows

ToolLIFT: Lifting Tool-Specific Trajectories into Function-Level Graphs for Generalizable Tool Planning

Xiuhui You, Jiayi Luo, Zichao Shen et al.　·　arxiv: 2608.03468

Links: arxiv · alphaxiv

TL;DR

Abstracts tool call trajectories into "function-level workflow graphs," enabling agent tool planning experience to transfer to previously unseen tool sets, consistently outperforming existing methods across three OOD benchmarks.

Read Priority

Skim — must-read for teams with actual tool planning needs; skim the core idea if you're just using off-the-shelf tool chains.

Background

Existing tool planning methods (e.g., ToolBench, AnyTool) build dependency graphs between tools from historical trajectories. The problem: these graphs are bound to specific tool sets — switch to a different set of APIs and you have to relearn everything. This is especially painful in the MCP ecosystem, where tool sets change constantly.

Intermediate Guide

• Problem: Imagine you learned to plan trips using Google Maps + Uber. You move to a new city that only has Apple Maps + Lyft. If your experience is "open Google Maps then call Uber," it's completely useless. But if your experience is "look up a route then book a ride," it transfers immediately.
• Method: ToolLIFT first "lifts" each tool call trajectory to the functional level (search, convert, validate...) and builds a Functional Workflow Graph (FWG). During planning, it first determines the function sequence on the FWG, then maps each function to a concrete tool. Finally, it uses RL-trained source-gated rewards to ensure data flow between tools remains traceable.
• Why it matters: MCP makes dynamically changing tool sets the norm. Agents need not "memorize how to use these tools" but rather "understand the abstract structure of workflows."

Key Details

• Two ID benchmarks + three OOD benchmarks: consistently outperforms SOTA
• OOD transfer is the core selling point — can plan with tool sets never seen during training ⚠️ (author-reported, awaiting external replication)
• Clean architecture with decoupled function abstraction + tool selection, easily reproducible
• Medium barrier to adoption: requires defining a function category taxonomy; trajectory→function lifting currently relies on LLM
• Compatible with LangGraph / CrewAI tool orchestration layers — can be inserted as a planning module
• Limitation: function category granularity is currently semi-manual (human or LLM defined); a universal cross-domain taxonomy is an open problem

Reviewer's One-Liner

The "function-level abstraction" insight is clean and powerful, and the OOD experimental design is sound. But the function taxonomy definition is still semi-manual, which limits full automation.

Your Take-Away

• If your agent platform needs to support dynamic tool sets (MCP scenarios): ToolLIFT's "Functional Workflow Graph" is currently the most concrete transfer-based tool planning design
• If you're building tool usage evaluations: include OOD tool set testing — ID-only testing will seriously overestimate actual transfer capability

Today's Takeaway

I used to think the main risk of agent self-improvement was "not learning enough." Today's papers reveal the real risk is "learning the wrong things and growing more confident about it." Memory Reward Inflation and RoMeRL demonstrate from both theoretical and engineering perspectives that memory systems need more than just "store" and "retrieve" — they need active countermeasures against bias accumulation. ToolLIFT adds the reminder that the value of experience lies not in specific steps but in abstract structure — a principle that applies equally to tool planning and memory management.

References

• arxiv:2608.00017
• arxiv:2608.02508
• arxiv:2608.03468