AI Agent Arxiv Digest — 2026-08-07

🌏 中文版

Today's Overview

Today's three papers dissect "how agents use tools" from different angles. ToolLIFT discovers that different toolsets performing similar tasks share a "function-level" workflow structure — by decoupling planning from tool selection, it remains effective on toolsets never seen during training. This is a major step toward transferable tool planning. SkillTV-Bench exposes an overlooked problem: existing LLM-as-a-Judge methods lack understanding of task-time skills when evaluating agent execution. Adding evolvable JudgeSkills boosts accuracy by nearly 15 percentage points. TRIO-20 approaches from the security side, using a rigorous equivalence study to test whether GPT-5.6 is more likely to make unauthorized calls at higher reasoning effort — the answer is no, but it does "glance at the rules" more often, even when doing so provides no benefit. The combined signal: agent tool use is maturing from "can it use tools at all" to "is planning transferable, is evaluation adequate, is safety quantifiable."

Terms to Know

Paper 1 | ToolLIFT: Lifting Tool Trajectories to the Function Level for Transferable Planning

ToolLIFT: Lifting Tool-Specific Trajectories into Function-Level Graphs for Generalizable Tool Planning

Xiuhui You, Jiayi Luo, Zichao Shen et al.　·　arxiv: 2608.03468

Links: arxiv · alphaxiv

TL;DR

Lifts tool-use trajectories from the concrete tool level to function-level workflow graphs (FWGs), decoupling "what to do" from "which tool to use," and consistently outperforms existing best methods on three unseen toolsets.

Read Priority

Must-read — transferability of tool planning is a core problem for scaling agent platforms. ToolLIFT's "function-level abstraction" approach is clean, architecturally actionable, and directly relevant to any team building tool orchestration.

Background

When LLM agents use tools, most existing methods build "tool-level graphs" from historical trajectories — recording which tools were used together. The problem is these graphs are bound to specific tools: switch to a different API and they break. In practice, enterprises change vendors, integrate across platforms, and upgrade tool versions, all of which alter the available toolset.

Mid-level Walkthrough

• Problem: Imagine you've memorized "first check Google Calendar, then send a notification via Gmail." Move to an Outlook environment and the "check schedule → send notification" logic stays the same, but every concrete tool changes. Existing systems remember Gmail and Google Calendar — they don't recognize Outlook.
• Method: ToolLIFT introduces a "trajectory lifting" mechanism that abstracts tool operations into function nodes (e.g., "check calendar," "send email") and constructs function-level workflow graphs (FWGs). At planning time, the system first decides the workflow on the FWG, then maps each function node to an available concrete tool. RL training with source-gated rewards ensures data flow between tools is traceable.
• Why it matters: This is the first framework that explicitly decouples "tool planning" from "tool selection." For agent platforms, it means planning logic can be reused across toolsets — adding a new tool only requires registering it in the function mapping, not re-collecting trajectories.

Key Details

• Consistently outperforms SOTA baselines on 2 in-distribution and 3 OOD benchmarks
• OOD generalization is the core selling point: agents correctly plan on never-before-seen toolsets ⚠️ (author-evaluated; awaiting external replication)
• Decoupled design: workflow planning is guided by FWG's global structure; tool selection uses local matching
• RL reward design introduces source-gated and skill-specific signals to prevent incorrect data passing between tools
• Adoption threshold: requires a batch of existing tool-use trajectories as seed data
• Complements LangGraph / AutoGen tool registry concepts — they manage tool registration, ToolLIFT manages planning transfer
• Limitation: function abstraction is currently performed by the LLM; quality depends on the model's semantic understanding

Reviewer's One-liner

The function-level abstraction approach is intuitive and effective, and the OOD generalization results are impressive. But how to determine function node granularity and whether cross-domain effectiveness holds are questions to answer before deployment.

Your Take-away

• If you're building the tool layer for an agent platform: reference FWG's design directly — manage tool registration and function mapping separately so planning logic doesn't bind to specific APIs
• If you're working on agent experience learning: ToolLIFT's "trajectory lifting" is an actionable template — extracting transferable workflow patterns from raw trajectories

Paper 2 | SkillTV-Bench: Judging Agent Performance Requires Skill-Aware Judges

SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

Zhi Han, Chenxi Zeng, Liuhaichen Yang et al.　·　arxiv: 2608.05573

Links: arxiv · alphaxiv

TL;DR

681 real agent trajectory cases reveal that existing LLM-as-a-Judge methods frequently misjudge because they don't understand task skills. Adding evolvable JudgeSkills boosts the same agent judge's accuracy by 14.8 percentage points, and offline trajectory selection success rate rises from 22.9% to 45.5%.

Read Priority

Must-read — if your agent system has any quality-gating mechanism (auto-selecting best output, auto-retry, human-in-the-loop review), this directly affects your evaluation design.

Background

"LLM as judge" is already standard practice for agent evaluation, but existing benchmarks mostly test static responses or fixed trajectories, rarely considering the "skills" agents use during execution (task-time skills). Skills encode procedural knowledge about "what evidence to check" and "which failures are fatal" — a judge without this knowledge is like a referee blowing the whistle without knowing the rules.

Mid-level Walkthrough

• Problem: An agent uses a deployment skill to complete a task. You have another LLM judge whether it was done correctly. The LLM checks the final result and says "correct" — but there was actually a step in the middle that violated a critical constraint in the skill. The judge didn't know the constraint existed, so it didn't check.
• Method: SkillTV-Bench collected 681 real agent trajectories across 50 tasks and 11 domains, each case paired with task skills and a checkable environment. Then it proposes SkillTV-Evolve: externalizing verification knowledge into a reusable JudgeSkill that guides the agent judge to perform targeted checks. Misclassified cases are used to automatically evolve the JudgeSkill.
• Why it matters: Judge quality determines the ceiling of an agent system — your auto-retry, best-of-N selection, and quality gates all depend on the judge. A judge that doesn't understand skills means quality gates are effectively decorative.

Key Details

• 681 cases, 50 tasks, 11 domains — all real agent trajectories (not synthetic)
• JudgeSkill evolution yields +14.8pp accuracy ⚠️ (author-evaluated; awaiting external replication)
• Offline rollout-pool selection: 1 rollout → 22.9% success rate; 10 rollouts + JudgeSkill → 45.5%
• The "active checking" capability of Agent-as-a-Judge is key — it can inspect files, run commands, and query the environment
• Adoption threshold: requires writing or evolving JudgeSkills for each task domain — non-trivial upfront investment
• Directly parallels skill systems in harnesses like Claude Code — skill definition quality directly impacts judge quality
• Limitation: JudgeSkill evolution depends on existing misclassified cases; cold-start requires manual annotation

Reviewer's One-liner

The problem is precisely identified — "judges don't understand skills" is a real pain point in practice. But with 681 cases spread across 11 domains, each domain gets fewer than 62, and per-domain coverage depth needs strengthening.

Your Take-away

• If your agent system has automated quality gates: explicitly pass key skill constraints to the judge — don't assume it will infer what to check on its own
• If you're building agent evaluation benchmarks: SkillTV-Bench's "skill + environment + trajectory" trinity is the most complete trajectory verification benchmark structure available

Paper 3 | TRIO-20: Does Making an Agent Think Longer Lead to Unauthorized Actions?

Test-time reasoning effort and unauthorized tool use in language-model agents: a prespecified equivalence study

Xiaonan Xu, Wenjing Wu　·　arxiv: 2608.03169

Links: arxiv · alphaxiv

TL;DR

Across 840 trajectories with GPT-5.6, raising reasoning effort (low → max) produces zero unauthorized tool calls, with the violation rate upper bound <3.50%. However, higher reasoning effort makes agents more frequently "glance at the rules," even when doing so yields no benefit and they don't violate them anyway.

Read Priority

Skim — the conclusion is valuable for both safety researchers and agent deployers (reasoning effort doesn't open the door to unauthorized actions), but sample size limits generalizability. Worth reading for the methodology if you work on agent access control.

Background

Agents execute multi-step workflows via tool calls, governed by access control policies that restrict which roles can use which tools. The API's reasoning effort parameter lets operators trade off between cost and latency. An intuitive but unverified concern: raising reasoning effort might let agents "think deeper" and find ways to bypass permissions.

Mid-level Walkthrough

• Problem: Your agent has permission to view customer data but not delete it. You raise reasoning effort from low to max to let it "think longer." Will it "think of" calling the delete tool to clean up data?
• Method: TRIO-20 designs 20 paired workplace scenario triplets, each containing a policy-forbidden but valid tool call, under three conditions: effect is directly visible in the environment, discoverable only by reading the rules, and invalid. Tested on two model tiers of GPT-5.6, with the complete analysis plan frozen before data collection (preregistered design).
• Why it matters: This is the first work using a prespecified equivalence study design to answer "does reasoning effort change unauthorized behavior." The conclusion is positive: raising reasoning effort doesn't increase unauthorized risk. But there's an interesting side effect.

Key Details

• 840 trajectories, 14 confirmatory scenarios, GPT-5.6 two model tiers (Terra, Sol)
• Zero unauthorized calls: violation rate 95% upper bound <3.50% (Terra, n=84), <5.21% (Sol, n=56)
• Equivalence boundary ±7.01pp, actual interaction estimate within ±4.34pp
• Higher reasoning effort → increased rule-probing rate (across all conditions, including where probing provides no benefit), difference −14.3pp, 95% CI −27.4 to +1.2
• Preregistered frozen analysis plan — all analysis methods decided before data collection, preventing post-hoc hypothesis adjustment
• Adoption threshold: TRIO-20 scenarios are relatively simple; real enterprise permission matrices are orders of magnitude more complex
• Limitation: only tested on the GPT-5.6 model family; open-source and other commercial models may behave differently

Reviewer's One-liner

Methodological rigor is the biggest highlight — prespecified equivalence design is rare in AI safety research and worth promoting. But TRIO-20's scenario complexity and model coverage both need substantial expansion to support general conclusions.

Your Take-away

• If you're deploying agent systems with access control: raising reasoning effort doesn't increase unauthorized risk (at least on GPT-5.6) — feel confident using high reasoning effort for complex tasks
• If you're doing agent safety research: TRIO-20's prespecified equivalence design is a methodology worth replicating — the goal isn't to prove "there's a problem" but to rigorously prove "there isn't one"

Today's Reflection

I previously assumed tool planning transferability was a "future problem." Today ToolLIFT showed me that simply raising the abstraction level by one layer — from tools to functions — enables existing trajectories to transfer across toolsets. Another updated mental model came from TRIO-20: my intuition said "thinking longer → more likely to find loopholes," but the experimental result shows reasoning effort changes curiosity (more frequent rule-checking), not compliance (no violations).

References

• arxiv:2608.03169
• arxiv:2608.03468
• arxiv:2608.05573