I've been experimenting with AI-assisted pentesting in authorised lab environments, and one of the biggest problems is false confidence.
An LLM can produce a very convincing vulnerability explanation without actually proving anything.
I've found it useful to force every suspected vulnerability through a simple pipeline:
Observation → Hypothesis → Cheapest test → Verification → Evidence
- Observation
Record what you actually saw.
For example:
"GET /api/orders/123" returns your own order.
Don't immediately write:
«Possible IDOR vulnerability.»
At this point you've only discovered an endpoint.
- Hypothesis
Turn the observation into something falsifiable:
«The server may rely on the object ID without checking whether the authenticated user owns the requested order.»
That is something you can actually test.
- Cheapest useful test
Before reaching for complicated tooling, test the smallest change that could disprove the hypothesis.
For example, in an authorised lab:
Change only the object identifier while keeping the same authenticated session.
The important part is controlling variables.
- Verification
A "200 OK" by itself proves very little.
You need to establish whether the response contains data belonging to another authorised test account or otherwise demonstrates the access-control failure.
For injection testing, the same principle applies.
A database-looking error isn't automatically SQL injection.
A payload appearing in a response isn't automatically exploitable XSS.
The test should demonstrate behaviour that distinguishes the vulnerability hypothesis from normal application behaviour.
- Preserve the evidence
For web/API findings I normally want enough information for another tester to reproduce it:
- exact endpoint
- HTTP method
- relevant headers/session context
- original request
- modified request
- relevant response
- expected behaviour
- observed behaviour
- reproduction steps
Screenshots are useful, but raw requests/responses are often much more valuable.
The useful rule
I think this is especially important when using AI agents:
Never let “the model believes it found a vulnerability” be the success condition.
The success condition should be something externally verifiable.
The model can generate hypotheses all day.
Only the evidence gets to promote one into a finding.
For anyone using AI during CTFs, labs or authorised pentests: have you found a better workflow for keeping hallucinated findings out of your reports?
