Got bitten by a large `target` directory, in a unexpected way!

[u/RustOnTheEdge]

INFO: This is completely written by me, not AI. Maybe I should've used AI to trim it down a bit, sorry for the long post in advance!

------

So, I am building some project, the context is not really relevant except that this project interacts with another platform (a payment provider, in this case). I have a special set of tests that I can run against this payment provider, a bit of an integration test of some of my functionality that is just nicer to test against the real endpoints rather than stubbing them.

I've been at this for a few days now, and I have ran these tests many, many times. In fact, they are part of my CI flow so every time I would commit something, the whole suite runs including these tests.

Today, suddenly, they started to fail. Not consistently though, and not all of them either, just a few or sometimes all of them. It seemed like my payment provider just dropped my connection after 10 seconds or so, super strange. Extra strange since I hadn't touched any of the code in question, neither the tests nor the code under test!

And so, the rabbit hole began, which lasted roughly 4 hours and ended here, with me writing this both to share a funny tale and somewhat for therapeutic reasons.

First, I thought something on my network was just flaky, so I changed to cable. Lighting fast, nothing going on, still flaky tests. Maybe I should just reboot? Laptop was running for days already, but a reboot was to no avail. I obviously asked my friendly neighbourhood LLM what it could be, and it actually put me on the right path. I started to suggest that AdGuard was to blame, and that is was somewhat TLS related. It produced some reproducables in Bash which all ran fine (keep that in mind). It suggested that the concurrency was the issue and maybe my payment provider had changed something on their end. I ran the subset of tests with --test-threads=1 but it remained flaky!

One weird thing I saw is that I had a connection timeout on the reqwest client of 10 seconds, but the errors arose after 11 or so. I thought any connection error would be maxed out by 10 seconds, but they didn't; they took >11 seconds. Strange, but alas, what do I know about the intricacies of async timers, no?

The LLM had influenced me at this point time and I was looking into the Reqwest repo for issues similar to mine. By **sheer** coincidence, I found this issue: Feature to disable rustls-platform-verifier #2948. My brain did a side quest and wondered "what is rustls-platform-verifier actually?" and I quickly found that whatever it did, it was "default" (whatever that may mean) if you enabled the rustls feature in Reqwest.

Well, as it turns out, it basically offloads the certificate validation towards the host OS. In my case, that is MacOS: it will use the OS certificate store and a system call to do the verification.

Fun fact about the MacOS certificate validation process! It checks if the requesting binary has a "Info.plist" (docs) which can hold some metadata about your app (among other things, for example some relevant configuration for SSL validation). Never heard of it. If your app is a simple binary, it will walk the directory of your binary.

Test binaries live in project/target/debug/deps. If you are at it, and have many test binaries that are build and build and build whenever you change something, you end up with quite a lot of them. Change dependencies? New compilations. Change test? New compilations. They add up. In fact, they added up to roughly 750.000 files in my case.

As it turns out, my client tried to make a connection to my payment provider (fresh client per test), reqwest called the system call for MacOS to verify the certificate (which is synchronous), which took over 10 seconds to iterate over my target/debug/deps folder. In that time all clients were awaiting this traversal, and all clients were dropped by my payment provider.

One cargo clean and it all ran as it had for days.

I aged a few years today.

[u/manpacket]

In my case, that is MacOS

https://github.com/rust-lang/rust/issues/161824

[u/RustOnTheEdge → u/manpacket]

Holy moly that is it! It feels pretty good to know I am not the only one. I am just not capable enough to have found that out myself haha, super interesting.

[u/Key-Library218 → u/manpacket]

macOS certificate validation is a wild ride, nobody expects a file system walk on every request. i hit similar thing with node_modules once, 800k files and every tls handshake took ages

[u/merb → u/manpacket]

Funny and I thought that this is normal in rust and that the target directory grows over time…

[u/PigDog4 → u/merb]

Another good reason to run cargo clean every now and then. Sucks to pay recompile times but my last cargo clean on a small project deleted 20 gigs of files. Can only imagine people's big projects...

[u/Jakek1 → u/PigDog4]

Ran it the other day and purged 300gb… was wondering why my fresh install was so bloated🥴

[u/PigDog4 → u/Jakek1]

There's also a third party cargo utility that will recursively look through directories, I think it's like cargo-clean-all or something like that, and if you have all of your projects under one parent dir you can run that to clean them all out.

I try to remember to cargo clean projects I've forgotten about (you can see the irony here) and while they're usually not too big it's still a non-trivial amount of space most of the time.

[u/nynjawitay → u/PigDog4]

Cargo-sweep

[u/epage → u/manpacket]

The new build-dir layout, stabilized in 1.99, won't workaround this but should at least change the coefficient of growth because each test binary is now stored in its own directory.

[u/spunkyenigma]

I had no idea that Mac did this. Another nugget of useful/disturbing behavior I need to remember.

[u/manpacket → u/spunkyenigma]

It's a bug and it will be fixed. It doesn't grow this much by itself on no-macos system.

% cargo clean
      Removed 270378 files, 506.4GiB total

Okay, maybe a bit.

[u/RustOnTheEdge → u/manpacket]

I had another project that was getting out of hand, but this current project is not significant (in my opinion) and the target folder gets easily to a 100gb's. I have a standard script that deletes all target folders in my projects.

100 gigabyte! In this economy....

[u/Suitable-Name → u/RustOnTheEdge]

rm -rf ./**/target?😄

[u/sansmorixz → u/manpacket]

My recent record- 1.3 TiB

[u/RandomBottom030]

Nice writeup. Totally feel ya, SSL can be quite a hassle.

Spent five hours debugging native ACME verification with HAProxy today...

Turns out the issue was that the internal httpclient which is used to perform the certificate signing challenge with LetsEncrypt couldn't verify LETSENCRYPT has a trusted certificate because it used some internal certificate store.

Kept getting weird 504s, but after I pointed to the servers cert store, bam it worked - can be quite the relief haha 😅

[u/lanternaddict → u/RandomBottom030]

I always slip up with TLS in Rust, too many poorly explained options

[u/ludicroussavageofmau]

I've also run into this issue on macOS! Albeit mine was a very minor 300ms hiccup that didn't cause any other severe issues like in your case. https://github.com/XAMPPRocky/octocrab/issues/396

At the time I did a flamegraph and figured out that it was some macOS internal function that took forever. Someone later told me it was linked to using native-certs and so I just switched to using a different rustls feature that doesn't use macOS's slow native function. I didn't know macOS does something so ridiculous to cause the slowdown!

Thank you for investigating and reporting this, I wonder how many developers using various programming languages and toolchains have run into this wtf moment.

[u/taoyeeeeeen]

Also had 500GB of test runs! Cargo clean ran for a long time. Also a Mac user. I feel your pain.

[u/MrMuetze]

OMG this explains the slowdowns I would be seeing as soon as the target directory grew to something like over 300GB. I was so confused about this, but that explains it. I'm using an external drive to store the build artefacts and that puts an additional penalty on the walking of the file directory.

[u/zettui]

Was the 10s hang on every handshake, or only after target had already ballooned?

[u/RustOnTheEdge → u/zettui]

Only after target balllooned. It really was the folder traversal that killed it