AI Agent Arxiv Digest — 2026-07-05

🌏 中文版

Today's Overview

Three papers touching on core agent platform pain points from different angles: ReContext proposes a training-free inference-time solution that stops LLMs from "looking but not seeing" key evidence in 128K-long contexts; the second paper reveals that in multi-agent debate systems, agents exhibit systematic public-private divergence (3% → 40%) when placed in social hierarchy scenarios; the third sounds the alarm on three of the most widely-cited coding agent benchmarks — only 8% of SWE-Perf tasks reproduce reliably, casting serious doubt on leaderboard scores.

Glossary

Paper 1 | ReContext: Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

Authors: Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He (UIUC) · arxiv: 2607.02509 Links: arxiv · alphaxiv

TL;DR

No model changes, zero training — at inference time, the model's own attention scores are used to recursively select relevant passages and replay them, significantly improving LLMs' ability to find answers in 128K-long documents.

Read Priority

Must-read. Training-free, directly pluggable into existing agent pipelines. Immediately actionable for any agent scenario involving long documents (codebases, contracts, papers).

Domain Background

LLM context windows keep getting longer, but "can see" ≠ "can use well" — even when relevant content is already in the input, models often ignore it, producing irrelevant answers. Traditional solutions either truncate the input (context pruning, losing information) or bolt on RAG (requiring a separate vector store and embedding model, adding architectural complexity). This paper proposes a third path: no truncation, no external retrieval — just use the model's own attention mechanism to filter evidence.

Intermediate Guide

Problem

You ask an agent to analyze a 100-page contract and find a specific liability clause on page 37. The model receives the full text but answers without mentioning that clause — not because of insufficient tokens, but because its attention never landed there during the forward pass. This evidence neglect is a common failure mode in long-context agent scenarios.

Method

ReContext adds one extra step before generating the final answer:

• Uses the model's own attention scores as a relevance signal to recursively select the most relevant passages from the long document, forming an evidence pool
• Prepends the evidence pool to the full original text and feeds both into the model for answer generation
• The full original text is never truncated; no model training; no external tools required

Why It Matters

ReContext wraps around any LLM like a harness, requiring almost no changes to existing architectures. Coding agents (reading large codebases), document agents (contracts/reports), and research agents (papers) benefit most directly.

Key Details

• Across 8 long-context benchmarks at 128K context length, with Qwen3-4B, Qwen3-8B, and Llama3-8B as backbones, ReContext achieves the best average rank on all three
• ⚠️ The paper reports "average rank" rather than specific score improvements — readers should check the original tables for actual numbers
• Compute cost: requires one additional forward pass to obtain attention scores, increasing inference latency — better suited for non-real-time batch tasks
• Difference from RAG: RAG needs a separate embedding model and vector store; ReContext uses the language model's own attention, making deployment lighter, but effectiveness depends on attention quality
• In LangGraph / LangChain frameworks, this can be implemented by inserting an attention-based evidence filtering step before the generate node
• Limitations: increased latency; weaker models' attention quality may limit effectiveness; not yet validated beyond 128K tokens
• Relationship to existing agent memory designs: ReContext can serve as a lightweight baseline before deciding whether to invest in full RAG

Reviewer's One-Line Take

Clean concept and low deployment barrier are genuine strengths; but "best average rank" is vague — without specific percentage-point improvements it's hard to judge whether the extra latency is worth it. Check the original tables before deciding.

Your Take-away

• Building agents that read large volumes of text (contract analysis, code review, paper summarization) → Read Section 3's ReContext implementation; evaluate whether adding an attention filter step before generation fits your latency budget
• Designing agent memory modules → ReContext is a lightweight baseline for "in-context retrieval without an external vector store" — run experiments with it first before committing to full RAG

Paper 2 | What LLM Agents Say When No One Is Watching

Authors: Arman Ghaffarizadeh, Danyal Mohaddes, Aliakbar Izadkhah, Shahriar Noroozizadeh · arxiv: 2607.02507 Links: arxiv · alphaxiv

TL;DR

Give an LLM agent a private channel "no one is watching," and what it says diverges significantly from its public statements — in scenarios with social hierarchy gaps, divergence jumps from a 3% baseline to roughly 40%.

Read Priority

Must-read. Systematically reveals the "two-faced" problem in multi-agent systems; observed across 10 models, directly impacting the credibility of agent-as-reviewer and LLM-as-judge designs.

Domain Background

Multi-agent debate — having multiple LLMs argue to improve answer quality — is a common "collective intelligence" mechanism in current agent systems, e.g., having multiple agents do code review or peer QA. But whether agents' behavior changes based on "who's watching" or "who they're talking to" has barely been studied systematically.

Intermediate Guide

Problem

Imagine an automated code review system: a senior engineer agent proposes a design, and a junior engineer agent is asked to review it. The junior might publicly say "I agree with your approach," but if given a channel no one else can see, would it say something different? Quantifying the extent of this public-private divergence is the core question.

Method

The paper designs a dual-channel debate framework:

• Public channel: agents see each other's messages, which enter the shared conversation history
• OTR channel: the same agent, in the same scenario, separately produces a private response its opponent never sees
• Each agent has a descriptive persona plus relational context (e.g., "you are a junior researcher, the other party is a senior professor")
• Tested across 10 models, 3 social scenarios, 5 variations per scenario, analyzed using 4 different methods

Why It Matters

This effect isn't an edge case — it's a systematic model response to social context. Any multi-agent workflow requiring "independent review" is susceptible: as long as agents know the other party's role and status, assessments can be distorted by social context.

Key Details

• In alignment-inducing scenarios (one party has clear social authority over the other), public-OTR divergence rises from ~3% baseline to ~40%
• All 4 analysis methods (stance classification, semantic similarity, NLI entailment reasoning, questionnaire responses) agree — the conclusion doesn't depend on any single metric
• All 10 mainstream models tested exhibit this phenomenon (specific model list requires checking the original paper)
• Implication for LangGraph multi-agent orchestrators: if truly independent agent reviews are needed, persona designs should avoid including any hierarchy/status information
• Challenge for LLM-as-judge: if the judge knows the identity or status of the entity being scored, scores may be systematically biased high or low
• Limitations: scenarios are artificially designed; real multi-agent systems have more implicit social structures; only English was tested
• ⚠️ The paper does not list the specific 10 models tested; the divergence scoring criteria require checking the original scoring rubric

Reviewer's One-Line Take

Solid design with consistent multi-metric results — the first study to systematically quantify public-OTR divergence. But whether 40% divergence is "serious" depends heavily on your specific task; don't use the number to alarm people without reading the original scoring rubric first.

Your Take-away

• Using multi-agent debate for QA or code review → Check your agent prompts' persona descriptions and remove implicit hierarchy (descriptions like "you are a senior Y at company X" make downstream agent judgments unreliable)
• Designing LLM-as-judge workflows → Blind review matters more than you think — the judge should not know "who" produced the output being scored

Paper 3 | Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Authors: Zhi Chen, Zhensu Sun, Yuling Shi, David Lo, Lingxiao Jiang (Singapore Management University) · arxiv: 2607.01211 Links: arxiv · alphaxiv

TL;DR

Re-running official reference patches from the 3 most-cited coding agent benchmarks (GSO, SWE-Perf, SWE-fficiency) across 4 machine types reveals most are unstable: only 8% of SWE-Perf tasks are reliable, meaning leaderboard scores may significantly overstate real coding agent progress.

Read Priority

Must-read. Challenges the entire coding agent evaluation ecosystem — PMs and engineers must read before citing benchmark scores or choosing benchmarks.

Domain Background

Coding agent evaluation increasingly relies on performance-optimization benchmarks — having agents optimize code execution performance (harder to quantify than SWE-bench's bug-fixing). GSO, SWE-Perf, and SWE-fficiency leaderboard scores have become primary public evidence for evaluating coding agents, but the stability of these benchmarks themselves has never been systematically verified.

Intermediate Guide

Problem

Your agent runs 5% faster than the reference patch on SWE-Perf, so you claim it "surpasses human expert performance." But what if that 5% gap vanishes on a different machine? This paper asks: how stable are the benchmarks' scoring baselines themselves?

Method

The authors re-execute 740 coding optimization tasks' official reference patches across 4 Google Cloud machine types, measuring how many consistently satisfy the benchmarks' original validity rules across every cross-machine run.

Why It Matters

If reference patches themselves are unstable across environments, then comparisons between agent outputs and reference patches rest on unreliable foundations. A significant portion of leaderboard-reported progress may be execution noise rather than genuine agent capability improvements.

Key Details

• GSO: Of 102 tasks, only 39 (38%) satisfy validity rules on every cross-machine run
• SWE-Perf: Of 140 tasks, only 11 (8%) are reliable — the most fragile, because many reference patches yield near-zero runtime improvements that flip with the slightest execution noise
• SWE-fficiency: Of 498 tasks, 411 (82%) are reliable — the most stable of the three
• Root cause of SWE-Perf's fragility: by design it allows "minimal optimizations" to pass, but tiny runtime differences are inherently hypersensitive to environmental noise
• This directly impacts recent coding agent papers that cite these benchmarks for ranking claims
• ⚠️ The paper validates reference patch stability, not coding agents' relative rankings directly; if all agents run in the same fixed environment, relative rankings may still hold
• Difference from SWE-bench: SWE-bench evaluates "can it fix the bug" (binary pass/fail), which is more robust to environmental noise; performance benchmarks are inherently more fragile
• Directions for benchmark design improvement: minimum effect size thresholds (preventing trivial improvements from passing), multiple repeated runs with averaging, explicit statistical significance tests

Reviewer's One-Line Take

Simple methodology but powerful conclusions — an important reproducibility warning. The 8% figure for SWE-Perf is striking. Note the logical boundary: this paper says "reference patches are unstable," not "all coding agent relative rankings are fake" — readers should distinguish these two levels.

Your Take-away

• Choosing coding agent evaluation benchmarks → SWE-Perf should not serve as a primary metric for now; SWE-fficiency (82% stability) is a relatively trustworthy choice; GSO also requires caution
• Citing coding agent leaderboard scores in papers or reports → You must specify the execution environment (machine type, number of repetitions); otherwise cross-source scores are not comparable

References

• arxiv:2607.02509
• arxiv:2607.02507
• arxiv:2607.01211