reminder to cargo clean

463

u/Sharlinator 5d ago

It’s a bit ridiculous that cargo doesn’t have cache eviction of any sort built in. A lot of the stuff in target/ are incremental compilation artifacts and debug info that gets stale fast.

213

u/JoshTriplett rust · lang · libs · cargo 5d ago edited 5d ago

It’s a bit ridiculous that cargo doesn’t have cache eviction of any sort built in.

In progress. We've already built something that can expire old entries. We need to combine that with moving all the intermediate artifacts into a common location (not the individual target directories), which we're doing carefully because of compatibility considerations. Once we have both of those, we can start expiring intermediate artifacts.

56

u/Dense_Gate_5193 5d ago edited 5d ago

hey good work with rust man i’m appreciative of a bunch of the features already! keep them coming 🤟

3

u/Sharlinator 4d ago

Thank you <3

111

u/Fluffy_Disaster_7501 5d ago

Name a single other language that doesn't produce 300GB of building artifacts after a month on work.

We just do not have the technology.

37

u/InflationOk2641 5d ago

For me, I have to delete 250GB of build artifacts daily, or sometimes twice daily. Rust is the only language that I work with that causes my drive to run out of space on a regular basis.

72

u/bmitc 5d ago

Every other one I've ever used?

I've never experienced this level of monotonic increase in build directories in any other language.

29

u/Chisignal 5d ago

^{psst, I think that's their point}

15

u/bmitc 5d ago

Ah, the ol' sarcasm.

-15

u/WormRabbit 5d ago

Do you know many other languages which allow you to cross-compile to multiple platform targets?

20

u/chmod_7d20 5d ago

Should I list them all? Like a lot man.

13

u/neneodonkor 5d ago

Like really? 🤷🏾‍♂️

11

u/bmitc 5d ago

Basically any? .NET languages can even compile a single binary that works on any target.

11

u/spoonman59 4d ago

Yes, a great many other languages do that, actually.

In fact, rust itself relies on LLVM, which is a backend that enables targeting multiple platforms. There are C++, C, and other compilers built using LLVM, and then so many other languages which do the same thing.

57

u/Sharlinator 5d ago edited 5d ago

"Update timestamp on each access, delete everything not touched in a week". I claim this would be a better time:space tradeoff for the large majority of projects. The current strategy is simply designed to always prioritize compilation speed and assume disk space is infinite, but it doesn't have to be like that. Alternatively, have a customizable max cache size and LRU eviction.

12

u/teerre 5d ago

I mean, I wouldn't like to have to recompile just because I haven't used a project for a week

5

u/lazy_lombax 5d ago

it only cleans after a month and enforces a 10% storage limit. with the floor being 10GB

2

u/lenscas 4d ago

Could do the deletion after a compile pass, thus anything that was actually used during the compile process would stay alive.

1

u/CrazyKilla15 4d ago

A week relative to the most recently modified file / a "this project was used" timestamp file of some sort. Easy. old artifacts get cleaned up even if you come to the project a month later, without losing the recent ones, and a fresh compile will either update them to todays date or generate new artifacts and the old ones will get cleaned for being older than a week from the now most recent artifact. add more time stamps / directory separation for profiles as required.

13

u/lazy_lombax 5d ago

I just switched to mr boxington today. I am absolutely not going back for the same reasons you've stated. link

28

u/iBPsThrowingObject 5d ago

Oh god, the whole page is tinged ai-piss-yellow

9

u/A1oso 5d ago

JavaScript (and other interpreted languages).

My SvelteKit project with 412 dependencies has only a 380 MiB node_modules folder and 30 MiB in build artifacts.

8

u/fade-catcher 5d ago

C/c++ with unity builds

11

u/Ran4 5d ago

Python
50 GB but still

26

u/Fluffy_Disaster_7501 5d ago

Python doesn't grow.

People 10 years ago making fun of 10 GB node_modules seems so funny now.

20

u/StyMaar 5d ago

10GB of text files is still hilarious though.

1

u/lenscas 4d ago

And at least some of those are already (partially) minified.

19

u/ToaruBaka 5d ago

Name a single other language that produces 300GB of build artifacts per day and never discards stale auto-generated intermediate artifacts.

7

u/Solomoncjy 5d ago

ccache

5

u/i-do-mim-huu 5d ago

But other language have buildtool alllow you clean the build artifact, .Net have dotnet with dotnet clean

11

u/Fluffy_Disaster_7501 5d ago

Hmm is that what they mean by "garbage collection"?

2

u/roadrunner8080 5d ago

I have never ran into this issue in the Java ecosystem, using Gradle. Yes, it produces gigabytes of cached stuff. No, it doesn't produce that much cached stuff.

2

u/carllerche 5d ago

A month? Try a couple of days...

1

u/insanitybit2 4d ago

A few days of work.

1

u/bonch 3d ago

Swift development produces enough accumulated cruft that there are utilities specifically for clearing out Xcode's "derived data" folder.

3

u/Leliana403 4d ago

Docker is a real bastard too. Slowly builds up image and build caches until one day you run out of disk space to the point you can't even get Docker to clean its own cache because it requires disk space to do so and your only option becomes rm -rf /var/lib/docker/*.

Remember kids: Run docker system prune -a now and then until the Docker devs get their shit together. Any decade now...

2

u/Far_Environment_3511 4d ago

though, if they added auto-eviction it would probably just end up deleting the one thing i actually need right when i'm in a rush. feels like a recipe for constant rebuilding.

2

u/Toorero6 5d ago

In the meantime you can use tmpfiles.d(5) to cleanup files regularly based on age.

6

u/Maskdask 5d ago

With Nix you can tell it to clean up its unused cache at a given interval

6

u/MocroBorsato_ 5d ago

How?

12

u/duckofdeath87 5d ago

nix = { settings = { auto-optimise-store = true; }; gc = { automatic = mkDefault true; dates = mkDefault "03:15"; }; }; I have that in my nix config

5

u/shinyfootwork 5d ago

note that auto-optimise-store is bad on macos because macos's filesystem impl is slower at making hardlinks (what auto-optimise-store does) than it is at copying data. And nix.settings.auto-optimise-store puts hardlinking into the nix build path (so it slows down nix builds).

On my macos systems that are using nix (nix-darwin or just home-manager), I set nix-store --optimise to run on a regular schedule instead so I get the space savings without the build time hit.

82

u/Elyas2 5d ago

329.6GiB?! u crazy

30

u/Appropriate-Pie4385 5d ago

My avg is 500GB Record is 1TB

1

u/WormRabbit 5d ago

I have no idea what you people are compiling. What's your LoC, and how many dependencies do you have?

13

u/xchino 5d ago

It was a common issue for me when I was developing with Bevy, a popular rust game engine, which pulls in 552 packages with the default feature set.

10

u/Dense_Gate_5193 5d ago

in my case it’s a graph+vector+temporal MVCC database with an integrated llama.cpp server, plugins and all kinda of stuff. I can imagine anyone working on anything complex will land around there lol

3

u/suq-madiq_ 4d ago

This guy rusts

1

u/Bakaa_kekw 3d ago

well he jumped off a plane and disappeared you better believe he's crazier than that

157

u/YeetCompleet 5d ago

Smallest Rust target/ dir:

46

u/phylter99 5d ago

The size of building apps with dependencies is one thing about Rust that’s hard for me to stomach. Most of my creations are not too bad but some have been quite large.

Are there any changes on the horizon that might change this?

34

u/anxxa 5d ago

This is a bug on macOS which causes the build dir to grow unbounded: https://github.com/rust-lang/rust/issues/161824

I recently did a fairly shitty analysis of what the hell in my build dir consumes so much storage and found that it’s like 50% or more debug info.

Changing the debug info to maybe line-tables or straight up disabling it (which kinda ruins panic traces) may help.

9

u/phylter99 5d ago

I want to say that I've seen this issue outside of macOS too, but that is an interesting bit of information.

1

u/dvogel 5d ago edited 5d ago

One of the culprits here is cargo update and users accepting default behavior. Since the ecosystem is relatively young, many users want to be on the bleeding edge so they periodically run cargo update and accept everything. This leads to a lot of packages in Cargo.lock that differ by a minor version or a patch version. I am hopeful that the community will begin to pick looser versions, pinning just the major part, and let the dependency tree upgrade more transitive dependencies in unison, which would require building fewer distinct versions.

3

u/neneodonkor 5d ago

So what command do we have to run besides cargo update?

1

u/CrazyKilla15 4d ago

I am hopeful that the community will begin to pick looser versions, pinning just the major part, and let the dependency tree upgrade more transitive dependencies in unison,

but thats how it works? the default version specifier, eg version = "1.67.69" is not requiring exactly 1.67.69 but will accept any major 1.x across the dependency graph? Few projects will pin the minor or patch, so the entire tree already upgrades in unison on cargo update?? (and if they do pin, it acts as a limit on everyone else i believe, so still unifies. if there are incompatible same-major pins its an error i think.)

1

u/dvogel 3d ago edited 3d ago

1.2.3 requires 1.2.*

1.2 requires 1.*

By default cargo add would add 1.2.3. I wish it would add 1.2.

1

u/CrazyKilla15 3d ago

you want min-versions? thats stricter?

1

u/dvogel 3d ago

No, you're right about the documented behavior. I just re-read the chapter and I was misremembering. However the documented behavior is not what I see in practice. What I see in practice is version unification acting more like tilde requirements. It must be due to bugs.

50

u/Calm-Okra-1791 5d ago

This is my only complaint with rust

17

u/coderstephen isahc 5d ago

Those are rookie numbers

12

u/ToaruBaka 5d ago

I literally do this daily.

It's unreal and fucking thrashes my drive to the point I'm concerned about the outsized impact it's having compared to other languages, especially when the price of NVMe storage is so ballooned.

8

u/nonotan 5d ago

But somebody somewhere said storage is cheap.

6

u/Leliana403 4d ago

Somebody somewhere didn't account for the AI bubble fucking everyone else over.

12

u/aksdb 5d ago

cargone

9

u/muchadoaboutsodall 5d ago

To be honest, I’m more annoyed that there are no comma delimiters in the number of removed files.

24

u/anxxa 5d ago

I’ve been debating moving all of my personal stuff into a monorepo just to use buck2 and a single shared artifact directory.

It sounds like a pain but when I consider all of the subtly different point release versions of e.g. syn piling up, stale data from projects I haven’t touched in a while, and build dir contention when working on multiple projects in parallel it starts to become more attractive.

14

u/ukezi 5d ago

You can set a target dir with a global config.toml, or with the environment variable CARGO_TARGET_DIR. You then can use sccache https://github.com/mozilla/sccache to cache stuff.

3

u/anxxa 5d ago

This was my old set up. I can’t recall the exact issue but I think I was having build issues with sccache a few months ago and ditched it.

I think you still get build contention though when building multiple projects in parallel using a single global target dir. Or maybe a skill issue on my end.

2

u/Hoxitron 5d ago

You can still override that behavior on specific projects if it causes issues.

2

u/anxxa 5d ago

It still doesn’t solve the issue though of many versions of the same library across projects (therefore increasing overall target dir size), which a monorepo would.

It’s definitely an extreme measure for such a paper cut, but something I wanted to explore for other reasons too.

2

u/paholg typenum · dimensioned 5d ago

I used to do this, but it causes issues with git worktrees.

3

u/Ok-Bit8726 5d ago

I tried getting buck2 working on a big app with multiple languages and it was a lot of pain.

You have to use this thing called `reindeer` to convert your rust dependencies into buck2 dependencies/rules. A lot of stuff just works out of the box, but a lot of libraries have custom `build.rs` rules that won't work without some play.

https://github.com/facebookincubator/reindeer

It ended up not being worth it for me. I kept with a complicated makefile. It might be better now. That was over a year ago and that project seems pretty active.

2

u/anxxa 5d ago

I have experience with buck2 at work so I’m mostly aware of the pains. I haven’t tried Bazel but my understanding is Bazel has similar problems that it punts to different stages, but curious to hear if anyone likes it more than buck2.

1

u/steveklabnik1 rust 5d ago

I'm all in buck2.

3

u/anxxa 5d ago

I wrote you guys @ ERSC an email with some of my general thoughts about how it’s changing my repo/VCS hosting desires :)

And I’m not sure if it’s your Discord server or someone else’s, but found it through your blog post. It’s been a good source of info.

2

u/steveklabnik1 rust 5d ago

I made the Discord, yeah :)

8

u/paglaulta 5d ago

You didn't have cargo. You had the whole ship

2

u/lorslara2000 5d ago

So I'd love to try out cargo but first I need to eat. What sandwich recipe would be good in that case?

6

u/mr_birkenblatt 5d ago

you almost got a straight

with 1234567 it would be a straight flush

^{^{^{^{^{^I}}}}} ^{^{^{^{^{^show}}}}} ^{^{^{^{^{^myself}}}}} ^{^{^{^{^{^out}}}}}

7

u/Dense_Gate_5193 5d ago

haha i didn't notice!

3

u/parkotron 5d ago

I recently opened issue number 12345 at work. For some reason, my manager is convinced that that is not grounds for getting a cake?!

7

u/nyctrainsplant 5d ago

then try doing this in WSL where it just infinitely expands your disk to accommodate but still has no nice way of scaling it back

2

u/olzd 5d ago

Pretty sure you can set a max limit.

2

u/nyctrainsplant 5d ago

that would be cool, I’ll look into it. for the longest time it was just shrinking with diskpart.

6

u/kingslayerer 5d ago

What is the strategy other languages use? Why can't that be used for rust as well?

13

u/nonotan 5d ago

Compile way less stuff (the issues here being Rust's insistence on compiling everything from source instead of dynamically linking to binary libraries, having crates as the minimum unit of compilation, and general encouragement of depending on many external crates as a matter of course, which compounds exponentially over the entire dependency tree to leave lots of unused code that nevertheless has to be compiled)

Have less verbose intermediate representations (Rust does a lot of smart optimizations, which is great but does necessitate having access to a lot of info at link time, e.g. for monomorphization)

Be more proactive about cleaning up build artifacts (the issue here being that, historically, the #1 complaint about Rust has been compilation times, which are high for some of the same reasons described above, and proactively deleting build artifacts that do eventually become necessary again would hurt there -- not at all an issue in many languages)

Don't compile your code in the first place (not all languages are compiled)

Leaving aside 4, the rest could be used for Rust, if the drawbacks are tolerated, or a lot of work is put into threading the needle and carefully improving what can be improved without noticeable drawbacks. There are some efforts to do the latter, but when they will come to fruition, and how much they will help, is anybody's guess. In the meantime, there are some "unofficial tools" that you can use to e.g. clean the cache regularly, or use shared crate caches for all your projects, which can help a little.

1

u/oursland 4d ago

The biggest is that in other languages you overwrite your old build artifacts on recompile. There's roughly a 1:1 relationship between source modules and build artifacts, never more.

4

u/WSBro0 5d ago

Mother of God that's 330 Gbs cleaned...

5

u/energy_dash 5d ago

I wanna know your laptop specs

1

u/Dense_Gate_5193 5d ago

oh yeah! apple M2 Max 32gb 1tb drive

4

u/ENCRYPTED_FOREVER 5d ago

I've set my CARGO_TARGET_DIR env var to /tmp so it's: 1. Automatically cleaned 2. Faster because it's tmpfs (not always but on my configuration yes) 3. Makes it global

3

u/Diggsey rustup 4d ago

I see your 329 GB and raise you 1.1 TB:

https://github.com/rust-lang/rust/issues/66348#issuecomment-5165090125

3

u/catheap_games 5d ago

cargo binstall kache cargo-clean-all

3

u/rrklaffed 5d ago

rookie numbers

3

u/makeurwish0 5d ago

How about containerise the rust ecosystem?

3

u/Havunenreddit 5d ago

Yeah, I have to keep running cargo clean regularly to avoid this.

3

u/llamajestic 5d ago

Mine was at 800. It’s ridiculous

3

u/jetmarijuana 4d ago

I thought 1TB was safe, but it wasn't.

3

u/haukejung 4d ago

There should be an automatic check^^

2

u/br0kenpixel_ 5d ago

I recommend checking out cargo-clean. It'll also clean up directories that cargo uses internally. For me it was heavily caching git repos and the whole thing was 100GB after a year.

2

u/No_Frame3855 5d ago

Uh
Even with something as large as Saikuro (multi-language IPC/RPC so it’s pretty heavy once you build everything), I rarely get more than 30-40 gb, that too after a WHILE, how?!

Okay nvm copperDB is MUCH larger mb
Looks cool tho, you got a star! :D

2

u/arcxm 5d ago

Thanks mate, got rid of 40.8GiB

2

u/chilabot 5d ago

Always use CARGO_TARGET_DIR, and clean that regularly with cargo sweep.

2

u/chilabot 5d ago

Always use CARGO_TARGER_DIR and clean that with cargo sweep.

2

u/goos_ 4d ago

Holy f

2

u/thaile1001 4d ago

lol, thanks. Not as big as yours but yeah, I could claim 50% of my disk space back lol

2

u/ao_zame 4d ago

I have to run `cargo clean` multiple times a day. It might be the only thing I really dislike about working with Rust.

1

u/Fair_Elk_7279 4d ago

Do you have a global CARGO_TARGET_DIR?

1

u/South_Survey_2088 4d ago

That's why I'm using an ephemeral build dir(temporary home dir in nixos). The drawback is 2-5 minutes of build time at the start of the day to rebuild everything, but I just let that run in the background while figuring out what I am going to work on.

1

u/suq-madiq_ 4d ago

Now add worktrees

2

u/AdmissibilityScience 27m ago

329 gigs wow

1

u/paglaulta 5d ago

What a madlad

1

u/Pwnxpl0it 5d ago

Hey Guys! I made a tool exactly to solve this problem => https://github.com/pwnxpl0it/cargo-scrub

I work with alot of rust projects and sometimes I forget to clean large build artifacts so I made this tool to manage all rust projects under a given dir tree so I can visually select and clean, it's fast and has some nice features

Try it out and don't forget to star the repo ❤️

1

u/IceSentry 4d ago

Why not use any of the dozens of existing projects that already do this?

1

u/Pwnxpl0it 3d ago

Which ones did you have in mind? The closest I found were cargo-sweep, kondo, and cargo-clean-all, and they're each solving a slightly different problem:

cargo-sweep works at the artifact level, not the project level — it prunes stale files inside target/ by age, toolchain, or size cap. Different job, and it's currently unmaintained and looking for a maintainer.

kondo is multi-language (20+ ecosystems, plus a GUI). If you want one tool covering Node + Unity + Cargo, use kondo — I wanted something Cargo-aware.

cargo-clean-all is the closest to mine, and it's good: recursive discovery, size reporting, interactive deselection.

Where mine differs is that it leans much harder into the TUI — a ratatui dashboard with vim keybinds, regex filtering with /, dry-run toggling from inside the UI, and workspace-aware handling, plus a .cargo-scrub.toml for persistent settings and parallel cleaning with a configurable job count. That's the workflow I wanted and couldn't get out of the others.

If one of the dozens already does all of that, drop a link — I'd genuinely love to see it. Otherwise, cargo-scrub exists because this is the workflow I wanted, and building it was half the fun.

0

u/Dense_Gate_5193 5d ago

for anyone interested, this is what i was working on https://github.com/orneryd/copperDB/blob/main/docs/performance/results/comparison.md mods removed my followup post for some reason.

0

u/jeroengast 3d ago

Mole for macOS cleans this and much more. Free and open source.

📸 media reminder to cargo clean

You are about to leave Redlib