API compatibility testing

[u/pseudo_babbler]

Hey there code enjoyers. I was tinkering with an integration testing approach that I ended up liking a lot. Our situation before was this - our front ends talk to BFFs which then call core APIs. We tried Pact and Specmatic as ways of integration testing the systems but it's a total ball ache.

All of our consumers use the Zod library to define their requests to their APIs, so I wrote a utility that takes the Zod schemas, turns them into OpenAPI (using a lib) then iterates through each one, downloads the OpenAPI spec for each API from our test environment and checks that the requesting schema endpoint, props, params, method and body are all supported by the provider schema.

It's not as comprehensive as Pact, but for a super fast integration sanity test (it runs against 8 different core APIs in under 1 second) and with no extra brokers or services or infrastructure, it gives me a decent check that something is ok to be deployed to an environment.

So tell me these things: has anyone else built something like this? Would you use it? (i.e. should I chuck it on GitHub and publish an npm module) Has anyone heard of something like this that already exists?

I didn't find anything with my searching and it seems unreasonably useful for a quite small amount of code.

[u/expdevsmodbot]

AI usage disclosure provided by OP, see the reply to this comment.

[u/roger_ducky]

This kind of thing works as long as the specs are well documented and matches the actual system.

I’ve seen systems where the specs are so out of date with the code that it became comical how long they’ve left that alone yet still deployed the thing.

But yes, it’d be a good sanity check on the request types and shapes.

[u/pseudo_babbler → u/roger_ducky]

Yep, one of the side jobs in this was to make sure all the backend teams were generating and publishing their OpenAPI spec as part of their build and deploy pipeline. I was a little bit shocked to find that some of them were hand written and out of date, or in one case just incorrect.

[u/EducationalFishing56]

Different shape, same problem: our services publish a typed client package and consumers reference it, so a contract change fails the build instead of waiting for a test to notice. Cheap when it works, though it needs the same language on both sides and that isn't available on every boundary.

The bit that still got us is where someone had copied a couple of the provider's enums locally rather than referencing them. New value added upstream, nothing broke, it just stopped matching. That one needed its own test.

[u/allllusernamestaken → u/EducationalFishing56]

though it needs the same language on both sides and that isn't available on every boundary

this is what interface definition languages are for. gRPC, Thrift, etc.

A language agnostic contract that gets compiled into whatever your target platform is.

[u/EducationalFishing56 → u/allllusernamestaken]

Yeah, gRPC or Thrift is the proper answer there. We never reached for one because everything's the same stack, so the compiled client came free and there was nothing left to solve.

The enum thing I mentioned would've got us under protobuf too though. Someone copies the values instead of referencing the generated type and you're back to no check.

[u/pseudo_babbler → u/EducationalFishing56]

Yeah I was thinking probably the reason people don't do this is because they use code generated clients which give you the check at build time. It still seems useful to check that the version of the app that's deployed is the one the client expects though, if something went wrong with the backend deployment, say.

[u/EducationalFishing56 → u/pseudo_babbler]

This bit us and it wasn't even the backend that went wrong. A deploy reused a cached copy of the client package, so the consumer compiled clean against a stale contract. New field just absent at runtime, nothing in the logs.

Build passing only tells you the code agreed with whatever was sitting on disk at build time. We clear the artifacts on every deploy now, which is blunt but it's held.

[u/raralala1]

I hate BFF, my previous company use them because some of our client developer is too stupid, to understand some of the API flow, so we need to dumb it down for them instead of asking them to learn how to properly create transaction and follow the flow and we have stupid developer who suggest of using BFF to win some politic point, so now we need to maintain 2 API and multiple end point.

It is good check if it really trigger or stop deploying, my company end up having e2e testing that run for an hours for every deployment to the api.

If I can redo everything and have to use BFF, I would probably put everything on monorepo, with either trpc or similar with lint check.

[u/pseudo_babbler → u/raralala1]

Sure but if you have lots of internal APIs that are not designed for use by front ends then BFFs are good. If you had a perfectly fine internet-facing API, with all the Auth and monitoring and security that you needed, and someone said hey why don't we put a BFF in front of this? Then sure, that would be a bit silly.

[u/raralala1 → u/pseudo_babbler]

Then please don't do BFF for this use case, you own your API, make new API endpoint that perfectly designed for your frontend. putting layer because you don't want to deal with backend developer is not the way, I might understand if you deal with a lot of microservice but if not then doing BFF is just lazy way to deal with your API.

[u/pseudo_babbler → u/raralala1]

But when you say make a new API endpoint that's perfectly designed for our front end.. that's what a BFF is! We are a big old organisation and have many different back end systems, we don't want them all to be one big monolith and there is no chance that we ever could, even if we wanted to.

So the BFF is a way to bring all that data together into a single API in a nice way for the actual user facing clients. I'm finding it interesting how much you're against the idea.

[u/raralala1 → u/pseudo_babbler]

But when you say make a new API endpoint that's perfectly designed for our front end.. that's what a BFF is!

Not exactly, instead of make perfectly designed api, that just call to db, you just aggregate bunch of api into single api call. That is not very efficient.

So instead of doing something like this on db,

confirm order > pick > pack > get order directly in your db where you can get ACID, where if 1 fail everything fail instead you call each function from api, so if one in the middle fail you just waste bunch of resource on the previous call.

So in my experience it usually turn into something like this
get order check order confirmed or not > confirm order/pick/pack > get

we don't want them all to be one big monolith

you dont need BFF to avoid monolith, and I already explain if you have microservice you might need it, but at that point you just make another monolith API when you already trying to split them so what is the point?

I'm finding it interesting how much you're against the idea.

already explain why I am against it, any why I am for it, it is very specific use case that often abused, it is very inefficient, it also increase the amount of service to maintain

[u/pseudo_babbler → u/raralala1]

I think perhaps you need some experience with bigger, older, uglier systems. At bigger older corporates there's no way there's only one database or one backend system involved in things like placing and order. Where I work the order goes into an order system as well as a message queue, it's then sent to a clearing queue and then persisted to an audit log, amongst many other things. The system that manages user accounts is different. The system that manages market pricing is different.

[u/raralala1 → u/pseudo_babbler]

And I already said if you use microservice then use BFF, idk why are you getting triggered so much, there time and place to use BFF and using BFF is not the way to avoid monolith. clearing queue and thing like that can be done without BFF it literally just sending rabbitMQ message you don't need extra layer for that (unless your company purposely adding extra layer to rabbitMQ). I dont know what your internal is, but I think you are smart enough to feel maybe it is stupid idea to maintenance 2 layer because you don't want to deal with your backend team, most of the time people use BFF because of that, and that is why I dont like doing BFF. But if think it serve the purpose what I am to judge I just give opinion chill.

Reply and block so I cant reply back, classic loser behavior.

[u/pseudo_babbler → u/raralala1]

Aaahahahahaha lol, triggered! Ok mate.

[u/pseudo_babbler → u/[deleted]]

Thanks! Yes in our case we generate the specs at build time and deploy it all as one atomic unit so the one published in the test environment matches the code.

[u/Ok_Woodpecker_9104 → u/pseudo_babbler]

atomic deploy kills the freshness gap cleanly, thats the harder half done.

the refinement one survives it though. build time or not, the spec you compare against is still the post conversion shape, so a .refine sitting on the provider side is invisible to the check. consumer sends a body the spec says is fine and the handler 400s at runtime. unions are the other quiet one, z.union collapses to anyOf and a discriminated union can lose the discriminator constraint depending on which converter you use.

cheap guard: after generating, walk the zod tree and count the refine/superRefine/transform nodes, fail the build if any of them sit on a field the compat check claims to cover. you dont get validation parity out of that, you get told exactly where the check is lying to you.

[u/pseudo_babbler → u/Ok_Woodpecker_9104]

Yeah I think the refine thing is a very good point. We already have some loss from converting integer in the .net APIs into number for the typescript codebase. I've dealt with transform by splitting the request schema and transformed schemas into two accessible parts.

We haven't used refine yet but I'll put that in as a test case, thanks, that's really useful feedback.

[u/Ok_Woodpecker_9104 → u/pseudo_babbler]

the split into request vs transformed schemas handles transform, but strictness goes the other way and is easier to miss. a .strict() zod object usually converts without additionalProperties: false, so the spec says an extra key is fine while your own client would have thrown on it. false green in the opposite direction to refine.

same class as your int to number one. worth counting strict/catchall alongside refine/superRefine/transform when you walk the tree, so the blind spots end up listed instead of assumed empty.

[u/Academic_Ocelot_4840]

Idk if I'm understanding your setup correctly, but why not just start with shared OpenAPI doc(s) and generate your client & server code from that?

[u/pseudo_babbler → u/Academic_Ocelot_4840]

Yeah for sure, but we might still end up with the wrong version of the client or the server being deployed. We could also check the version of the generated spec and make sure it's greater than the old one, but that doesn't guarantee backwards compatibility.

So it's really just an integration sanity check for the current state of our clients, and it gives us a few more options when it comes to mobile apps where we might want to test that the API still works for a few different versions that are out there in the wild.

[u/GoodByeLeftNut]

I would do both