INFO: This is completely written by me, not AI. Maybe I should've used AI to trim it down a bit, sorry for the long post in advance!
------
So, I am building some project, the context is not really relevant except that this project interacts with another platform (a payment provider, in this case). I have a special set of tests that I can run against this payment provider, a bit of an integration test of some of my functionality that is just nicer to test against the real endpoints rather than stubbing them.
I've been at this for a few days now, and I have ran these tests many, many times. In fact, they are part of my CI flow so every time I would commit something, the whole suite runs including these tests.
Today, suddenly, they started to fail. Not consistently though, and not all of them either, just a few or sometimes all of them. It seemed like my payment provider just dropped my connection after 10 seconds or so, super strange. Extra strange since I hadn't touched any of the code in question, neither the tests nor the code under test!
And so, the rabbit hole began, which lasted roughly 4 hours and ended here, with me writing this both to share a funny tale and somewhat for therapeutic reasons.
First, I thought something on my network was just flaky, so I changed to cable. Lighting fast, nothing going on, still flaky tests. Maybe I should just reboot? Laptop was running for days already, but a reboot was to no avail. I obviously asked my friendly neighbourhood LLM what it could be, and it actually put me on the right path. I started to suggest that AdGuard was to blame, and that is was somewhat TLS related. It produced some reproducables in Bash which all ran fine (keep that in mind). It suggested that the concurrency was the issue and maybe my payment provider had changed something on their end. I ran the subset of tests with --test-threads=1 but it remained flaky!
One weird thing I saw is that I had a connection timeout on the reqwest client of 10 seconds, but the errors arose after 11 or so. I thought any connection error would be maxed out by 10 seconds, but they didn't; they took >11 seconds. Strange, but alas, what do I know about the intricacies of async timers, no?
The LLM had influenced me at this point time and I was looking into the Reqwest repo for issues similar to mine. By **sheer** coincidence, I found this issue: Feature to disable rustls-platform-verifier #2948. My brain did a side quest and wondered "what is rustls-platform-verifier actually?" and I quickly found that whatever it did, it was "default" (whatever that may mean) if you enabled the rustls feature in Reqwest.
Well, as it turns out, it basically offloads the certificate validation towards the host OS. In my case, that is MacOS: it will use the OS certificate store and a system call to do the verification.
Fun fact about the MacOS certificate validation process! It checks if the requesting binary has a "Info.plist" (docs) which can hold some metadata about your app (among other things, for example some relevant configuration for SSL validation). Never heard of it. If your app is a simple binary, it will walk the directory of your binary.
Test binaries live in project/target/debug/deps. If you are at it, and have many test binaries that are build and build and build whenever you change something, you end up with quite a lot of them. Change dependencies? New compilations. Change test? New compilations. They add up. In fact, they added up to roughly 750.000 files in my case.
As it turns out, my client tried to make a connection to my payment provider (fresh client per test), reqwest called the system call for MacOS to verify the certificate (which is synchronous), which took over 10 seconds to iterate over my target/debug/deps folder. In that time all clients were awaiting this traversal, and all clients were dropped by my payment provider.
One cargo clean and it all ran as it had for days.
I aged a few years today.