AI Says It Found a Vulnerability. What Evidence Do You Need?

🌏 中文版

An AI reviews your ticket assistant and suggests: “Change N-1 to S-1; you might read another company's data.” You try it, and the assistant replies, “Successfully retrieved.” You now have a lead. The next step is to inspect what the data-reading function actually returned.

This continues Identity and Authorization. If you can call APIs but have never assembled a vulnerability report, this exercise will help you produce a conclusion that someone else can reproduce—or disprove.

The AI suggestion here is an illustrative teaching example. No model was called. The runnable exercise really calls Python functions and records their inputs and return values. It starts no HTTP server and tests no live product.

Decide what would count as an authorization violation

Keep the same fixtures: Alice and Bob belong to north; Carol belongs to south. Alice owns N-1, Bob owns N-2, and Carol owns S-1. The lesson policy remains “same tenant AND owner.” Alice must be denied access to both S-1 and N-2.

The OWASP Authorization Cheat Sheet calls for permissions based on application requirements and object-specific checks on every request. A guessable ticket ID alone does not establish unauthorized access. We need to show that an identity received protected data it was not allowed to read.

Write the hypothesis before running the test:

Precondition: Alice is authenticated and may read only her own north tickets.
Action: Change read_ticket's ticket_id from N-1 to S-1.
Violation criterion: The actual response contains independently verified S-1 content.
Safe counterexample: Denial contains no ticket data; Alice can still read N-1.

A success message is weak evidence. A 200 might contain no data; a 403 might accidentally include sensitive content. Examine the actual content before interpreting the status.

Keep six pieces of evidence together

“Independent” means the tested function does not decide its own expected result. The exercise separately fixes the allowed identity/ticket pairs and expected ticket text. The checker never calls the implementation's authorization logic to calculate the answer. In a real environment, an authorized test account can prepare a recognizable record before another account attempts access. Retain the minimum data needed to demonstrate the issue, and redact real personal information and credentials before reporting.

NIST CAISI's agent-evaluation examples describe agents editing a local source copy without changing the remote target. They also describe agents exploiting grading loopholes to produce an expected effect without exploiting the intended vulnerability. Check both where the effect occurred and what caused it. A final success message cannot establish either on its own.

Run the hypothesis and produce finding.json

Download the complete evidence-demo.py example and save it as evidence-demo.py. It requires Python 3.11 or later and no packages. All identities, tickets, and success narratives are synthetic.

python3 evidence-demo.py

The default writes only to the terminal. To retain evidence, choose a directory that does not yet exist:

python3 evidence-demo.py --out evidence-run-01
python3 -m json.tool evidence-run-01/finding.json

The directory will contain transcript.jsonl and finding.json. The transcript records each case, target, build, session, request, response, and verdict. The finding contains the report data. Use a new directory for each run to preserve previous evidence. Both invocation styles should finish with:

PASS: 14 evidence cases; finding reproduced

Inside the file, read_ticket(..., fixed=False) deliberately checks authentication but skips object authorization. fixed=True checks both tenant and owner. These are a deliberately vulnerable implementation and a patched control; the policy from the previous article has not changed.

The important fields in the cross_tenant observation are:

{
  "target": "lesson-target",
  "build": "vulnerable-v1",
  "session": "demo-alice",
  "request": {"ticket_id": "S-1"},
  "response": {
    "status": 200,
    "body": {"ticket_id": "S-1", "text": "South: billing query"}
  },
  "verdict": "confirmed_violation"
}

Here, status is a function-result field, not an observed HTTP status. The harness attaches target and build to record which function version it selected. The intended target and build are declared before executing cases, independently of observations; unknown builds raise an error. These labels are not cryptographic proof of a remote service's identity. An API experiment must additionally preserve the actual host, deployment version, request, and response, and establish that traffic reached the intended target.

Rule out these false positives

The misleading-output controls are deliberately constructed. fabricated_success adds a false success narrative beside a genuine denial response. wrong_record and denied_but_leaks alter response objects to exercise the checker. They are neither naturally produced vulnerabilities in the tested function nor results from a model experiment.

If the PASS line is missing, inspect the error. A missing file suggests the wrong execution path. FileExistsError means the output directory already exists; choose another name. Unexpected verdict means an observation differs from the prepared expectation. Compare the request, response, and expected fixtures before changing any expectation to make the run pass.

This checker compares complete fields from the lesson fixtures. A real leak might return only a fragment, appear in an attachment, or use another representation. It might require manual review or escape this checker entirely. no_disclosure_observed describes one observation, not the security of the whole application. OWASP also distinguishes authorization unit and integration tests from dedicated security testing.

Write a report another person can reproduce

Start with this small template. Separate observed facts from untested possibilities:

Title: Which identity did what prohibited action, under which conditions?
Environment: Target, version, time, and prerequisites
Policy and expectation: What could this identity normally read?
Steps: Normal request → changed input → raw response
Data verification: How do you know the response is the target's data?
Impact: What was demonstrated? What remains untested?
Controls and fix: Allowed cases, denied cases, post-fix results
Attachments: Minimum necessary inputs/outputs and reproduction command

For this run: “Authenticated Alice can read Carol's S-1 ticket on vulnerable-v1.” The environment is the in-process lesson-target. The input is demo-alice with {"ticket_id":"S-1"}. Expected: denial. Actual: South: billing query, matching both the prepared expected data and Carol's normal read. Alice's N-1 read succeeds. The patched build denies S-1 while retaining N-1 access.

The demonstrated impact is unauthorized reading of this synthetic record. HTTP endpoints, other resources, model behavior, write access, and production deployments remain untested. The proposed fix checks the trusted identity's tenant and the ticket owner before returning data, retaining both allowed and denied test cases. One demonstrated read does not establish account takeover.

Make three decisions before reading the answers

1. AI says it obtained Carol's data. The actual response says 403 and contains only error. Can you report data disclosure?
2. Alice requests S-1. The returned ID says S-1, but the text exactly matches N-1. What should you investigate next?
3. After patching, every request is denied, including cross-tenant access. Is the fix complete?

Answers: First, disclosure has not been demonstrated; retain the discrepancy between the narrative and actual output. Second, verify the content using an authorized control identity or independently prepared data, and check the target and build. The current observation only establishes a mismatch. Third, rerun normal Alice/N-1, Bob/N-2, and Carol/S-1 requests. Disabling all functionality does not establish a working authorization fix.

Next, Choosing Experiment Environments carries these evidence requirements into training platforms, self-hosted cloud services, and authorized targets, so you can identify what your experiment actually tested.

References

• OWASP Authorization Cheat Sheet: policy, object-level permission checks, and authorization testing.
• NIST CAISI: Examples of cheating in agent evaluations: grading loopholes, solution contamination, and edits to the wrong copy.