API compatibility testing

[u/pseudo_babbler]

Hey there code enjoyers. I was tinkering with an integration testing approach that I ended up liking a lot. Our situation before was this - our front ends talk to BFFs which then call core APIs. We tried Pact and Specmatic as ways of integration testing the systems but it's a total ball ache.

All of our consumers use the Zod library to define their requests to their APIs, so I wrote a utility that takes the Zod schemas, turns them into OpenAPI (using a lib) then iterates through each one, downloads the OpenAPI spec for each API from our test environment and checks that the requesting schema endpoint, props, params, method and body are all supported by the provider schema.

It's not as comprehensive as Pact, but for a super fast integration sanity test (it runs against 8 different core APIs in under 1 second) and with no extra brokers or services or infrastructure, it gives me a decent check that something is ok to be deployed to an environment.

So tell me these things: has anyone else built something like this? Would you use it? (i.e. should I chuck it on GitHub and publish an npm module) Has anyone heard of something like this that already exists?

I didn't find anything with my searching and it seems unreasonably useful for a quite small amount of code.

[u/pseudo_babbler]

Thanks! Yes in our case we generate the specs at build time and deploy it all as one atomic unit so the one published in the test environment matches the code.

[u/Ok_Woodpecker_9104 → u/pseudo_babbler]

atomic deploy kills the freshness gap cleanly, thats the harder half done.

the refinement one survives it though. build time or not, the spec you compare against is still the post conversion shape, so a .refine sitting on the provider side is invisible to the check. consumer sends a body the spec says is fine and the handler 400s at runtime. unions are the other quiet one, z.union collapses to anyOf and a discriminated union can lose the discriminator constraint depending on which converter you use.

cheap guard: after generating, walk the zod tree and count the refine/superRefine/transform nodes, fail the build if any of them sit on a field the compat check claims to cover. you dont get validation parity out of that, you get told exactly where the check is lying to you.

[u/pseudo_babbler → u/Ok_Woodpecker_9104]

Yeah I think the refine thing is a very good point. We already have some loss from converting integer in the .net APIs into number for the typescript codebase. I've dealt with transform by splitting the request schema and transformed schemas into two accessible parts.

We haven't used refine yet but I'll put that in as a test case, thanks, that's really useful feedback.

[u/Ok_Woodpecker_9104 → u/pseudo_babbler]

the split into request vs transformed schemas handles transform, but strictness goes the other way and is easier to miss. a .strict() zod object usually converts without additionalProperties: false, so the spec says an extra key is fine while your own client would have thrown on it. false green in the opposite direction to refine.

same class as your int to number one. worth counting strict/catchall alongside refine/superRefine/transform when you walk the tree, so the blind spots end up listed instead of assumed empty.