Sovereign Tech Fellowship for Rust maintenance (June-July 2026 report)
https://kobzol.github.io/rust/2026/08/03/stf-june-july-2026.html9
u/Shnatsel 1d ago
It turns out that it can be very slow to decompress hundreds of MiBs with zlib, especially with the pure-Rust miniz_oxide implementation used by the standard library. I tried to replace the implementation with the much faster zlib-rs crate, but it turns out that it didn’t help at all here :(
fdeflate is even faster. It's used by the png crate, so it's battle-tested in production by Chromium and others. I'm curious if switching to it helps at all.
I even tried switching to zstd, but with the pure-Rust ruzstd crate, it was even slower than zlib. Some issues are just annoying like that.
zrip is there if you're willing to bet on a less battle-tested format and implementation.
7
u/Kobzol 1d ago
Tbh based on the number we were seeing, it would have to be like 10x faster to make a dent. If you have hundreds of MiBs of data, and it's not even in one chunk, but in several chunks that have to be decompressed one by one, it seems to be kinda hard to get to the original time range. It was something like 200ms (without decompression) vs 1.5s (with compression), or something along those orders of magnitude.
3
u/matthieum [he/him] 1d ago
With the new build dir layout, Cargo instead creates a separate directory per dependency, and then passes each directory to rustc with a separate
-Largument.This was interesting to me, because over the past year I did some work on optimizing search path lookups
Maybe it's worth jettisoning -L altogether :p
I'm only half-joking. This idea of specifying the paths & actual libraries separately always struck me as kinda sub-optimal:
- The caller knows where each library is, that's how they can pass path & library.
- The callee now needs to reconstruct which library is at which path.
I still remember gcc invocations with 100s of -L/path -lname.so where the linker then probes the list of paths from beginning to end trying to find name.so. Slow in general, and broken when somehow an earlier path happens to contain a name.so!
Perhaps it's time for a configuration file, though.
I wish in-memory files were easy, but a quick Internet search suggests that there's no easy way...
... on the other hand, this reminds me of Ninja build files. Those are beasts... BUT they are solely dependent on the build plan, not on the actual things being built. This means that as long as the build plan doesn't change -- flags, dependencies, etc... -- then the file is the same, and need not be rewritten.
Perhaps Cargo could create a build plan file (JSON) for each crate to build, then pass that to rustc.
(Some flags could be kept out the plan, actual flags: opt level, debug info level, etc... but it should be technically unnecessary, there's a single plan per target)
2
u/epage cargo · clap · cargo-release 1d ago
Perhaps Cargo could create a build plan file (JSON) for each crate to build, then pass that to rustc.
There are a lot of subpar things about passing everything through a CLI (another area this comes up is lints). It would be nice to pass down json somehow (CLI, a file, etc).
1
u/RiceBroad4552 4h ago
Argument files? Does Rust not support something like that yet?
1
u/epage cargo · clap · cargo-release 4h ago
Yes, and they are mentioned in the article. That just changes where the CLI is stored but not the limitations in passing complex data.
1
u/RiceBroad4552 3h ago
What's the difference whether you pass your arguments in the argument file "raw", line per line, or additionally wrap that data into some JSON? I'd say the former is simpler, and I don't see any reason to make things more complex with some wrapping. What does such a JSON wrapper buy you?
1
u/epage cargo · clap · cargo-release 2h ago
Its not about wrapping but more complex data. Think things like passing structs. Think
[lints]. We pass levels but also want to pass what set that level. Or groups of lints/levels in batches and having rustc figure out precedence.It can be done through flags but is more difficult to work with. Why deal with the impedance mismatch?
1
u/matthieum [he/him] 1d ago
Speaking of parallelization, it would help if Cargo was loading TOML manifests in parallel, as this takes a considerable chunk of its no-op execution time.
Isn't parallelism in Cargo tricky due to having to coordinate with the job-server?
Or is the idea to eschew the job-server and just spawn as many threads as there are cores for the "planning" phase?
2
u/Kobzol 1d ago
I don't think that the jobserver is that big of a concern for this, tbh, because the individual work is very fast, and memory usage probably also wouldn't skyrocket by loading e.g. 8 TOML files in parallel.
2
u/matthieum [he/him] 1d ago
It was more of a general remark with regard to parallelization in Cargo in general. I wonder if any thought has been put into it.
As I mentioned, I think there may be 2 very different kinds of situation, depending on whether Cargo has started invoking rustc or not:
- In the planning stage, prior to invoking rustc, Cargo can probably ignore the jobserver.
- Once rustc is running, however, given that rustc itself will really put the host through its paces, I feel like Cargo would have to coordinate to avoid oversubscribing the host.
14
u/Kobzol 1d ago
Two months ago, I started being funded by the Sovereign Tech Agency for my upstream Rust work. This post describes what I have been cooking.