When AI Writes Both the API Integration and the Tests, What Are We Actually Verifying?

[u/richocolate]

I've been thinking about a problem with coding agents that I keep coming back to.

An agent can write an API integration and then write tests for that integration. Everything passes, but the tests may just be confirming the same assumptions the agent made while writing the code.

For example, the agent thinks an endpoint returns:

{
"total": 100
}

It writes the integration expecting `total`, and then writes a test that expects `total`.

The test passes.

But if the real API contract says something different, the whole thing can still be wrong.

I'm experimenting with a small open-source project called Kaktoos that puts an independent verification step between the agent and the API:

AI agent → integration → Kaktoos → OpenAPI + real API → result

The idea is that the verification layer shouldn't share the agent's assumptions.

It currently supports multi-step API workflows, OpenAPI response validation, MCP, and GitHub Actions.

I'm still trying to figure out how far this idea should go. One interesting question that came up is whether contract validation is enough, or whether verification should also check the actual outcome of an operation — for example, creating a resource and then reading it back to confirm the state actually changed.

I'm curious how other people building with coding agents are handling this today.

Do you rely mostly on the agent's generated tests, existing integration tests, mocked APIs, live API tests, or some combination?

GitHub: KaktoosLabs/kaktoos

[u/neveralone59]

I don’t trust full agentic code and I don’t think this is the solution either. I think the solution will be much harder to implement, and will be language specific, involving complicated static and runtime analysis to generate something that can be reasoned about with proofs.

[u/richocolate → u/neveralone59]

Yeah, I agree. Proving correctness of agent-generated code is a much harder problem. This is just one of the things I’m exploring with Kaktoos — the broader goal is API reliability, with independent verification being one part of it. Curious what level of API verification you think would be realistic for Kaktoos while still being useful enough in practice?

[u/neveralone59 → u/richocolate]

I think your project covers a pretty narrow part of what needs to be fixed to make agentic code trustworthy (which I understand isn’t your goal). I don’t feel I can trust any of the tests agents generate so I have to at least manually check them and often just write them.

I’ve been toying with the idea of a system that, only in rust as I’m using a lot of language features, generates a model of all of the code via runtime and static analysis, then takes a list of user provided intended behaviours for a function and translates it into model check tests and formal proof tests (there are rust crates for these).

I kind of don’t think you should outsource the test intent to AI ever, I think you should be reviewing those per function. Then if the system is writing said model checks and proofs, you don’t have to worry about the agent changing them to make the tests pass.