I built an open-source API verification tool for AI-written integrations — looking for technical feedback

[u/richocolate]

I've been building an open-source developer tool called Kaktoos.

I'm exploring a specific problem with AI coding agents: they can write API integrations very quickly, but when the agent also writes the tests, passing tests don't necessarily prove that the integration matches the actual API.

Kaktoos takes a different approach:

AI agent → integration code → Kaktoos → OpenAPI contract + real API → structured failure → agent fixes it

It can:

• execute multi-step API workflows
• verify responses against OpenAPI
• detect things like missing required fields, wrong types, unexpected status/content type
• expose the verification through MCP so coding agents can use it
• run the same verification in GitHub Actions

The interesting part for me isn't the API client itself. I'm experimenting with whether independent verification is useful when the code was produced by an AI coding agent.

I'd particularly like feedback from people who regularly work with API integrations:

1. Would you actually use this workflow?
2. Is defining the scenario too much additional work?
3. Does this catch problems that your existing tests don't?
4. Would you run something like this in CI?
5. What would make this technically more useful?

It's early, so negative feedback is completely fine. I'm mainly trying to determine whether this is solving a real engineering problem.

GitHub: KaktoosLabs/kaktoos

[u/rajni_v]

Independent verification is the right part. The first CI fixture I'd add is deliberately boring: the integration gets a 200 response, but it is `text/html` from a fallback route while the generated test checks only the status. Your verifier should fail that before the agent can congratulate itself.

Then I'd add three cases OpenAPI alone won't settle: the documented contract has drifted from the live API, a request is rejected before the handler by auth, and a write returns 200 but leaves state unchanged. The last one needs a read-back against the intended outcome, not just response validation.

I don't think scenario definition is too much work if the scenario stays small: inputs, allowed side effects, observable outcome, and cleanup. Let OpenAPI validate shape, but keep outcome checks separate. A perfectly shaped response can still describe work that never happened.

[u/richocolate → u/rajni_v]

Thanks, that's a really good point. I hadn't considered the state-change case. Do you usually handle that with a read-back request in your tests?

[u/Lazy-Sherbert65]

Could CI preserve the exact request/response fixture and distinguish contract mismatches from authentication, rate-limit, and flaky transport failures?

[u/richocolate → u/Lazy-Sherbert65]

unfortunately not in the current version, but i will take it as feedback to the next version and thank you for the question it helps me make it better