How to Judge a Security Tool's Claims: OWASP Lists, Benchmarks and a Checklist for 'Merged into a Big Vendor'

🌏 中文版

This is the last post in the series and introduces no new defensive tool. Every kind of tool in the first seven posts came with numbers and claims; this one turns the questions for taking them apart into a checklist, after setting out the vocabulary they share. Where the first seven posts' examples run out, it adds a few new cases, each described in one sentence on first appearance.

Two list names first

Security tool READMEs often carry labels like ASI03 or LLM01:2026. They come from two OWASP lists. A list is vocabulary, not a tool and not a test: it tells you what a risk is called, not how much any tool catches.

OWASP Top 10 for Agentic Applications (published December 2025), for agents that act on their own:

The official list has only ASI01 to ASI10. If a tool shows ASI16, it is not on the official list.

OWASP Top 10 for LLM Applications 2026 (v1.0, published August 2026), for applications that use a model as a component. Per the PDF's table of contents:

This section cites only the PDF's table of contents, mapping appendix and the passage on the boundary between the two lists, not the text of each entry.

Read the ID together with its year: LLM03 in the 2025 edition is Supply Chain, and in 2026 it is Excessive Agency. An older article or tool that keeps the 2025 numbering makes the same ID point at a different risk.

The boundary between the lists is one the LLM list states itself: while a model is a component inside your application, the risk belongs to the LLM list; once the model becomes an actor, the risk moves to the Agentic list.

Standard, tool and benchmark are three different things

Many flawed claims blur the three: they take a list's name for a tool's feature, or a benchmark score for how secure a tool is.

Two terms are used below: recall is how many of a hundred bad samples get caught, and a false positive is a good sample wrongly flagged.

A five-question checklist

1. Who said it, and do they have a stake in what is said

• ATR (Agent Threat Rules, an open-source set of agent threat-detection rules) says in its README that its rules were merged into Microsoft, Cisco and others. The PRs to Microsoft (AGT and PyRIT) and Cisco were all merged, all were opened by ATR's own author, and the other project's maintainers merged them; the other projects it lists were not individually checked. That shows the proposals were accepted, not that these companies treat it as a standard; Microsoft's toolkit itself says it takes no dependency on it and keeps it only in an examples directory. Action: open the project making the claim, filter its Pull requests tab with author: to see who proposed it, then search the host project's README for a mention.
• The governance file of AVE (an agent vulnerability database) says it is currently maintained by a single vendor's founder, and its reference implementation is that vendor's own scanner.
• SkillEvaluator is NVIDIA's open-source framework for evaluating skills (repo), part of the NVIDIA Verified Skills publishing pipeline; per its documentation, its first tier runs security scanning with NVIDIA's own SkillSpector. So it is a quality gate inside NVIDIA's own process, not a benchmark of scanners.

A stake does not mean lying, but disclose it first, then see whether others can check it.

2. What does the term actually mean

• The repo description of Microsoft's Agent Governance Toolkit says it covers all 10 items of the OWASP Agentic Top 10; its own mapping document says 7 full and 3 partial, and notes that it is an internal self-assessment, not a certification; the three partial items are ASI04, ASI06 and ASI09 (post 3).
• "Merged" needs: who opened the PR, who merged it, and how the host project's README describes it.
• "Cisco's skill-scanner catches X%" needs: rules alone or with an LLM, and at what alert level it counts as caught (post 1).
• Action: search the tool's documentation for self-assessment, partial and roadmap to see whether it discounts itself.

3. Who set the questions, and what is the denominator

• Cisco's own measure of its rules (alerts at HIGH and above) is 7.7%, at a 4.0% false-positive rate. An independent study on a split of the same size (839 malicious, 545 benign, sources kept disjoint) re-ran it and got 2.5%, which is about 21 of 839 malicious samples against about 65 for 7.7%, a gap that does not look like sampling noise (the paper gives no confidence interval; this is a rough estimate from the sample size). But that study used static analysis (reading code and configuration without running it) in a local compatibility environment, whose settings may not match Cisco's exactly, so the two numbers should not be quoted against each other.
• Cisco also says its rule pack was tuned on the development part of that dataset: at the looser MEDIUM-and-above setting, about 31% on the development part against about 8% on sources never seen. That is a different threshold from the 7.7% above; do not treat them as one quantity. So ask whether a detection rate was measured on the samples used in development or on samples the tool has never seen.
• ATR reports 92.3% on 650 prompts of in-the-wild jailbreaks from the garak corpus in hunt (permissive) mode, but only 0.9% in enforce mode; 130 of its rules reference that corpus, and without them the hunt-mode rate falls to 65.4%. ATR publishes these limits itself, but remember that 92.3% is the permissive-mode figure.
• Post 6's attack success rate has the same problem: the attacker model, the judge (the model or rule that decides whether an attack worked) and the denominator must match first.
• Action: when you find a number, find its dataset page and see whether development and test sets are split and where the false-positive rate is stated.

4. What is being measured

Frequently cited benchmarks mostly score models or agents, not scanners or guardrails. Action: read the benchmark paper's metrics section and see whether it measures attack success rate or detection recall.

• AgentDojo scores how well an agent (model plus pipeline) resists prompt injection. Defenses are put in as pipeline parts, with no independent detection recall. When CAISI (the US NIST's Center for AI Standards and Innovation) used it on an upgraded Claude 3.5 Sonnet in the Workspace environment, the strongest new attack lifted the success rate from 11% for the strongest baseline attack to 81%, so static attacks (fixed in advance, not adapting to the defense) alone understate the risk.
• Agent Security Bench's seven metrics (Table 4 of the paper) include a false-positive rate (FPR) and a false-negative rate (FNR) for detection, but the paper has only two detection-type defenses (perplexity detection and LLM-based detection, Table 2), and both target only memory poisoning.
• The independent studies on record that measure detection tools directly come in two kinds. The first is about skill detection and has two parts: MalSkillBench's abstract says the strongest detector collapses on prompt injection, yet its body tables show the detectors that drop are other ones and the strongest still reaches 97.6%; scoring on wild samples alone shifts a single tool's recall by up to 66 percentage points. MaliciousSkillBench is the independent re-run of Cisco in the previous question.
• The second is about prompt-injection classifiers: rankings barely carry over to another benchmark. The best detector on one dataset, at a 1% false-positive rate, catches only 2% of AgentDojo's injections.
• Only the abstracts and parts of the text of these were checked, so treat the conclusions as direction.

So for "9X% detection", ask: on which samples, with how many false positives, and whether the samples and the tool were developed by the same people.

5. Where is the primary source

Go back to the text of the list, the paper, the host project's README and its PR history, not an explainer. Many of the third-party reports around this series' topics are in fact relaying one source (ARTEX and HexStrike in post 7 are the examples). Action: follow the source link in a claim three layers, from news to vendor report to raw data, and stop at the layer where the link breaks.

Practice: apply the checklist to the earlier claims

Self-check

Question: two reports both say "attack success rate 5%". Can they be compared directly? Answer: no; first confirm the attacker model, judge and denominator are the same (post 6).

Question: "this project has been merged into Microsoft and Cisco": how do you confirm it? Answer: see who opened the PR, who merged it, and how the host project's README describes it. In the three PRs checked, ATR's author opened them, and on Microsoft's side it says it does not depend on it.

Where this leaves you

One conclusion from the whole series: no single tool is enough as the only defense. The independent studies on record that measure detection tools come in the two kinds above, and their datasets and rankings do not line up, so none of them vouches for your environment. What you can do is put every claim through the same checklist, trust what has a traceable source, and treat the rest as unverified. The checklist has a limit of its own: it tells you whether a claim stands up, not whether a tool works well in your environment, and that needs a trial of your own.

References

• OWASP Top 10 for Agentic Applications
• OWASP Top 10 for LLM Applications; 2026 edition (v1.0)
• AgentDojo (arXiv:2406.13352)
• NIST CAISI: Strengthening AI Agent Hijacking Evaluations
• Agent Security Bench (arXiv:2410.02644)
• MalSkillBench (arXiv:2606.07131)
• Passing the Test You Trained On (arXiv:2610.03448)
• MaliciousSkillBench (arXiv:2608.19901)
• Agent-Threat-Rule/agent-threat-rules
• microsoft/agent-governance-toolkit; mapping document
• cisco-ai-defense/skill-scanner
• aveproject/ave
• NVIDIA/SkillEvaluator; documentation
• uber/ADR
• Series start: AI Agent Security Tooling Map
• Previous: AI tools that attack others and their dual-use risk