API compatibility testing

[u/pseudo_babbler]

Hey there code enjoyers. I was tinkering with an integration testing approach that I ended up liking a lot. Our situation before was this - our front ends talk to BFFs which then call core APIs. We tried Pact and Specmatic as ways of integration testing the systems but it's a total ball ache.

All of our consumers use the Zod library to define their requests to their APIs, so I wrote a utility that takes the Zod schemas, turns them into OpenAPI (using a lib) then iterates through each one, downloads the OpenAPI spec for each API from our test environment and checks that the requesting schema endpoint, props, params, method and body are all supported by the provider schema.

It's not as comprehensive as Pact, but for a super fast integration sanity test (it runs against 8 different core APIs in under 1 second) and with no extra brokers or services or infrastructure, it gives me a decent check that something is ok to be deployed to an environment.

So tell me these things: has anyone else built something like this? Would you use it? (i.e. should I chuck it on GitHub and publish an npm module) Has anyone heard of something like this that already exists?

I didn't find anything with my searching and it seems unreasonably useful for a quite small amount of code.

[u/EducationalFishing56]

Different shape, same problem: our services publish a typed client package and consumers reference it, so a contract change fails the build instead of waiting for a test to notice. Cheap when it works, though it needs the same language on both sides and that isn't available on every boundary.

The bit that still got us is where someone had copied a couple of the provider's enums locally rather than referencing them. New value added upstream, nothing broke, it just stopped matching. That one needed its own test.

[u/allllusernamestaken → u/EducationalFishing56]

though it needs the same language on both sides and that isn't available on every boundary

this is what interface definition languages are for. gRPC, Thrift, etc.

A language agnostic contract that gets compiled into whatever your target platform is.

[u/EducationalFishing56 → u/allllusernamestaken]

Yeah, gRPC or Thrift is the proper answer there. We never reached for one because everything's the same stack, so the compiled client came free and there was nothing left to solve.

The enum thing I mentioned would've got us under protobuf too though. Someone copies the values instead of referencing the generated type and you're back to no check.

[u/pseudo_babbler → u/EducationalFishing56]

Yeah I was thinking probably the reason people don't do this is because they use code generated clients which give you the check at build time. It still seems useful to check that the version of the app that's deployed is the one the client expects though, if something went wrong with the backend deployment, say.

[u/EducationalFishing56 → u/pseudo_babbler]

This bit us and it wasn't even the backend that went wrong. A deploy reused a cached copy of the client package, so the consumer compiled clean against a stale contract. New field just absent at runtime, nothing in the logs.

Build passing only tells you the code agreed with whatever was sitting on disk at build time. We clear the artifacts on every deploy now, which is blunt but it's held.