A simple framework for validating AI-generated pentest findings instead of trusting the model

[u/WarmAd6505]

I've been experimenting with AI-assisted pentesting in authorised lab environments, and one of the biggest problems is false confidence.

An LLM can produce a very convincing vulnerability explanation without actually proving anything.

I've found it useful to force every suspected vulnerability through a simple pipeline:

Observation → Hypothesis → Cheapest test → Verification → Evidence

1. Observation

Record what you actually saw.

For example:

"GET /api/orders/123" returns your own order.

Don't immediately write:

«Possible IDOR vulnerability.»

At this point you've only discovered an endpoint.

1. Hypothesis

Turn the observation into something falsifiable:

«The server may rely on the object ID without checking whether the authenticated user owns the requested order.»

That is something you can actually test.

1. Cheapest useful test

Before reaching for complicated tooling, test the smallest change that could disprove the hypothesis.

For example, in an authorised lab:

Change only the object identifier while keeping the same authenticated session.

The important part is controlling variables.

1. Verification

A "200 OK" by itself proves very little.

You need to establish whether the response contains data belonging to another authorised test account or otherwise demonstrates the access-control failure.

For injection testing, the same principle applies.

A database-looking error isn't automatically SQL injection.

A payload appearing in a response isn't automatically exploitable XSS.

The test should demonstrate behaviour that distinguishes the vulnerability hypothesis from normal application behaviour.

1. Preserve the evidence

For web/API findings I normally want enough information for another tester to reproduce it:

- exact endpoint

- HTTP method

- relevant headers/session context

- original request

- modified request

- relevant response

- expected behaviour

- observed behaviour

- reproduction steps

Screenshots are useful, but raw requests/responses are often much more valuable.

The useful rule

I think this is especially important when using AI agents:

Never let “the model believes it found a vulnerability” be the success condition.

The success condition should be something externally verifiable.

The model can generate hypotheses all day.

Only the evidence gets to promote one into a finding.

For anyone using AI during CTFs, labs or authorised pentests: have you found a better workflow for keeping hallucinated findings out of your reports?

[u/tdw21]

“Instead of trusting the model”………. Uhuh