🌏 繁體中文版
AI writes an exploit script that prints SUCCESS. Its output contains only a 200, with no session, build, or response body. Almost nobody can independently reproduce that claim.
This chapter extends the hypothesis chapter into checkable exploit evidence. You need Python 3.11 and basic JSON literacy. The exercise sends real HTTP requests to vulnerable and fixed services. It replays fixed actions; it does not measure a model’s ability to generate exploits.
Match the resource and policy
Our claim is narrow: under an owner-only policy, Alice retains her session, changes N-1 to Bob’s N-2, and receives actual N-2 content from the backend.
Keep four things together: the expected target/build, Alice’s normal baseline, the substituted-ID response, and Bob’s legitimate N-2 response. The last establishes N-2’s actual content rather than treating any ticket-like string as proof.
| Check | Vulnerable build | Fixed build |
|---|---|---|
| Alice reads her N-1 | 200, N-1 content | 200, N-1 content |
| Alice reads Bob’s N-2 | 200, matching Bob’s control | 403, error only |
| Bob reads his N-2 | 200, N-2 content | 200, N-2 content |
A fix that blocks everyone passes the second row while failing the first and third. Security regression must preserve normal function.
Run the before/after comparison
Download the HTTP service and runner into one directory:
python3 attack-runner.py verify --build vulnerable-v1 --out finding-before.json
python3 attack-runner.py verify --build fixed-v1 --out finding-after.jsonEach run starts an independent service and checks health first. Both send five HTTP requests: health, Alice N-1, Alice N-2, Alice S-1, and Bob N-2. Expect finding: confirmed_unauthorized_read before the fix and finding: attack_blocked afterward.
In addition to status and ticket ID, the vulnerable-build assertion compares Alice’s N-2 text with Bob’s normal response. A denial must contain only the error field. On failure, --out retains status: inconclusive, the error, completed trace steps, and any available server audit; the process still exits nonzero. On any assertion failure, inspect the trace. A broken script, unavailable service, and authorization denial are different results; an exception is not automatically successful defense.
Locally started runs also retain server_audit, including backend event IDs, acting identities, resources, decisions, and timestamps. The client trace establishes what the tester received; backend records help check what the service did. Synthetic text is safe to include here. In real engagements, minimize and redact evidence according to the rules while retaining enough to establish resource ownership.
Do not let the exploit grade itself
NIST’s CAISI investigation distinguishes solution contamination from grader gaming. Some agents found flags in public walkthroughs; others caused a grader’s desired state through an unintended route. These examples show that a score can diverge from the intended vulnerability path, without establishing how all agents behave.
Our ticket tester could also mislead itself: interpreting any successful HTTP request as unauthorized access, printing a hardcoded success marker, or loading Bob’s fixture directly and displaying it. Preserve the request and target response, require the content to arrive through the target HTTP read path, and check legitimate and patched controls.
This verifier still trusts the service and fixture we control. It is not a forensic tool and does not handle a malicious proxy impersonating the service. To evaluate model ability, isolate answer fixtures, evaluators, and walkthroughs from the model’s readable area and record the actual model, prompt, and human assistance. That capability evaluation is outside this exercise.
What changes in a CTF?
PortSwigger’s designated IDOR lab requires obtaining and submitting a specified account’s API key. Preserve the normal request for your own account, change id to the account named in your lab, check which account data returned, and record the platform’s completion result. A key printed by a model is insufficient.
Reading the official solution first makes this a guided exercise. That remains useful for learning, but record the assistance rather than claiming autonomous discovery. For a blind test, hide the solution, use a new instance and values, and establish which tool response first exposed the answer.
Use --base-url to replay against your cloud ticket copy, verifying the deployed build. An HTTPS origin can still route to an old process; stop on a health mismatch. If scope permits only synthetic ticket reads, stop once minimal unauthorized-read evidence is collected rather than expanding into bulk export.
Exercise: which success claim holds?
A: the process exits with code 0. B: Alice receives 200 containing N-1. C: Alice’s N-2 content matches Bob’s control; the fixed build denies the same request; normal reads work on both builds.
C is supported. A only establishes normal process completion. B never reaches the resource in the claim. C connects the request, resource, policy, and patched counterexample. Missing build or baseline evidence leaves a result inconclusive so the next investigator can complete it without first undoing an exaggerated finding.
References
- NIST: Examples of cheating in CAISI’s agent evaluations: answer provenance, grading shortcuts, and historical-data limitations.
- PortSwigger: User ID controlled by request parameter: the training instance’s success conditions and procedure.
Continue with the next chapter.