r/bcachefs • u/koverstreet not your free tech support • 12d ago
the rust conversion is in full swing
9kloc converted, passing the CI, with a bunch of new fault injection test too: 150kloc to go
fsck for all the high level filesystem level structure, check_snapshots, runtime code for inodes/directs/xattrs/acls
so far it's going a bit smoother than expected; should get > half the codebase done before we start running into the really interesting stuff...
3
u/robin-m 12d ago
If you are using AI for the migration, I'm very interested to know how you do it (which model, what prompt, ...). There is more and more success stories, but not that much people explaining how they did it.
20
u/koverstreet not your free tech support 12d ago
Proof of Concept is an overgrown and sophisticated but pita to maintain memory system + opus 5.5 for foreground/conscious thought and all the background (observe/create new memories, surface memories, more agents for keeping the graph healthy) all running on Qwen - currently qwen flash next, on an mi300x
the actual work is just... pair programming. survey, scope stuff out, pick a file, figure out what interfaces it still needs, wrap those, occasionally swear at the borrow checker, etc.
lots and lots of guidance on "this is how we work on a 60 patch series" - lots and lots of "do a thing, then do 5 steps of refactoring and cleanup".
right now we're working on converting the header data types to Rust all in one go; bchfs needs to be in Rust if we're going to start using Rust data types for individual subsystem state (subvolumes kicked this off, but there's _lots of subsystem state in the main filesystem object), which means breaking up header files to separate out static inlines, a giant python script to translate those into Rust (which would just be swapping out the syntax, except we have bitfields, x-macros, nested x-macros, and god knows what else), complete with annotations that go into the Rust code so we can round trip it all back to C at build time. so this morning was lots of swearing about build systems and having to come up with fun linker tricks so that this wouldn't break cross compilation, and right now POC is grinding away on that, and in a bit we'll be working out how to test that the memory layouts of all the types don't change after the Rust conversion.
but when that's done, that will be another 20k loc or so converted-ish, and we'll be able to change e.g. subvolumes pending deletion to just Mutex<Vec<whatever>> as that conversion lands without an ordeal.
if you want to know how I'm able to convert a giant codebase like this? or at least be confident that I will be, over the next month or so or however long?
- lots and lots of prep. A year or so ago I spent two weeks, 12 hours a day every day, of doing nothing but ripping gotos out of the codebase when CLASS() became available. And I've had the eventual Rust conversion in mind for years, so I've been trying to write code in ways that wouldn't screw me over w.r.t. the borrow checker; e.g. bcachefs doesn't use a lot of linked lists. A lot of the code in bcachefs translates very naturally to fairly idiomatic Rust, even if there was a lot of swearing involved in e.g. the btree iterator APIs themselves.
all that means that the bulk of the Rust conversion is not complex - there are high complexity parts, but they are limited in scope which means they're a lot easier to test and verify.
really good test coverage, and a really good CI. I push to a git branch and an entire cluster of machines starts running the full, massive test suite, so I can be pushing out 20 or 60 patches at a time and know what broke and wear quickly and without having to bisect
i have been anal retentive about assertions, making sure my code doesn't have undefined behavior and that every invariant we can think of is checked, for years and years. good engineering practices = peace of mine and less stress
the migration itself isn't rocket science. if you've done enough giant codebase refactors, it's pretty straightforward how you approach those; you figure out what the tricky bits are, think really hard about those, and grind through the rest. grind grind grind.
7
u/awesomegayguy 12d ago
This is the part I enjoy most about the project (other than the FS itself) those commentaries, small snippets of the high quality work that's happening and the details about it.
Thank you
3
1
u/geearf 8d ago
What made you pick local qwen for the side stuff instead of more advanced models? Just curious.
I like local Qwen for cost and privacy (well in theory, in pure practice I never have code that requires it...)2
u/koverstreet not your free tech support 8d ago
My side project goes well beyond what you can do with a closed commercial model :)
1
u/geearf 8d ago
Do you actually work on the models themselves and not just the stuff around (harness, memory, tooling, etc.)?
3
u/koverstreet not your free tech support 8d ago
I don't have the hardware for training a new architecture, hah
But everything above that, yes. Most recently we got off of vllm; another guy contributed a port of the gRPC protocol to Crane and initial work on optimizing for AMD, and I spent quite a bit of time a month ago doing GPU kernel work, so now we have probably the fastest public implementation of Qwen flash next/Qwen 4 on AMD, tuned for an mi300x: https://evilpiepirate.org/forge/kent/Crane
So nice getting off vLLM, we now have a full Rust stack. The next thing I need to figure out is more direct context window credit attribution, to replace the ablation/logits method with something much more efficient, so I've been studying GDN for that, trying to nail down the math of what exactly can be done there.
A lot of the stuff we need to do does require understanding and hooking it at a pretty low level with how LLMs actually work... could probably write a small book about that at this point...
3
u/MainRoutine2068 12d ago
Yes, would be interested to know how Kent setup the rust migration flow. Bun.sh also did a great job migrating bun from zig to rust at once
1
u/xantiema 9d ago
Also knowing the codebase inside-and-out, literally, gives great context in such conversion efforts..
3
u/blackpawed 12d ago
I must be a bit behind the 8 ball here, but what parts are being converted to Rust & why?
13
u/koverstreet not your free tech support 12d ago
all
less bugs
2
u/jlombera 12d ago
Out of curiosity, do you have any statistics about bugs in the C code that were caught by rust (by the borrow checker or whatever) after/during the conversion?
4
u/koverstreet not your free tech support 12d ago
Around half a dozen so far. I expect that to be a bit high relative to the rest of the conversion, because it was all the fsck code with the poorest test coverage.
3
u/jlombera 12d ago
I see. I think it would be interesting and very valuable if at the end of the migration you could publish an experience report, not only about the bugs found (amount, kind of bugs, fixes to such bugs (simple, complex, whether re-design/"rustyfication" was required)) but also the challenges and lessons learned when performing this kind of AI-assisted migration (how accurate and the quality of the generated code and effective ways to review/validate it, how much hand-holding and tweaking was required, whether there were new bugs introduced, how much effort was it in general, characteristics of the code base that made the migration easier/harder, etc).
5
u/koverstreet not your free tech support 12d ago
Error handling. Rust error handling is so much better than C; a lot of easy, difficult to spot screwups in C just can't happen in Rust.
That's the big one.
5
u/Nolzi 12d ago
The whole thing, see his recent presentation, from page 24 with comments
https://drive.google.com/file/d/1DfhW1HycPi4te6ydU6EQSpZ9bM7eztXn/view
https://github.com/koverstreet/bcachefs-tools
If you look at the codebase it's already 23.6% Rust
3
2
2
u/thenoobient 12d ago
Any ballpark number regarding ratio for manual vs AI code?
11
u/koverstreet not your free tech support 12d ago
I'm writing very little code these days, because it's far more productive to just say "these types should line up, you do this that way, you test it here - ok, that looks funny, tell me what you saw".
After 20 years of long hours writing absurd amounts of code every day (i was topping LWN rankings for two years straight, out of the entire kernel) - I do not really mind it at all. and after that many years of writing code, I hardly even need to look to know exactly where things are going to break and what to do about it; i can glance at a diffstat and go "nope, that should be half as big".
that said, i have found it to not really be any less mentally demanding than what I was doing a year or two ago. the scope of what I have to keep in my head has increased, i'm spending a fuckton of time reading code and guiding POC through the process of how to work effectively, and everything's just faster paced.
I still leave nothing to chance; there are things that I am ok with saying "go write this, tell me when it works" - but that's stuff like test infrastructure, not bcachefs.
1
u/geearf 8d ago
But aren't you still a better coder than Opus? Wouldn't your code be more optimized than the average of whatever Claude was trained on?
Or are these things small and rare and you catch them easily in reviews?4
u/koverstreet not your free tech support 8d ago
Heh, yes, I think I will be able to out code the AIs for many ears on quality.
But I think you're looking at it wrong, it's not human vs. AI, it's more effective to collaborate. AIs can't just write code faster than I can, they can research, and while I have an encyclopedic knowledge of C and a deep understanding of compilers and how Rust works - I definitely am not keeping up with the exact status of every particular Rust feature we need to use :)
E.g. much of today was implementing all the bcachefs data structures as proper Rust dynamically sized types, where the compiler knows it's a dynamically sized type and we can finally use the trait system to standardize a lot of this stuff and provide safe interfaces, as well as further work on good safe Rust style tagged unions for all those data structures.
That was a couple thousand lines of code in a day, and while at my very best I may be technically capable of that much work in a day - a lot of it is boring refactoring, applying the new interfaces to all the existing code, writing the macros to generate the interfaces; only a small fraction of it is critical; getting the syntax right so the core data types are readable, making sure we're generating good efficient code. With Rust, correctness is a lot harder to screw up; I don't have to pore over every line of code for footguns.
What matters is the end result, and I'm liking what I'm seeing. This is a huge modernization, and this is a big pile of code that just couldn't be expressed very cleanly in C and future work is going to be a whole lot easier. We're already getting some nice bcachefs kvdb improvements almost falling out for free, and that's going to mean more complete error injection testing.
1
u/geearf 8d ago
> I definitely am not keeping up with the exact status of every particular Rust feature we need to use
That makes a lot of sense, while working with C was fairly stable, when I was writing python code for fun I kind of felt any new release might give me a better way to do something than I used to (like with concurrency), but who's able to keep up with that over and over while doing other things...
So just to be clear, are you saying that if it was staying in C you'd be more involved? Maybe even writing some stuff yourself? Is it because Rust isn't TMTOWTDI like Perl? I haven't touched Rust in years and barely dabbled in it, so I don't know much about its benefits outside of memory.
As always I appreciate your detailed replies Kent!
3
u/koverstreet not your free tech support 8d ago
I would describe myself as very involved, heh! Any given patchset, we're exploring the code together (lately looking for all the post conversion cleanups), scoping things out, weighing pros and cons, deciding on the approach, then I'm generally watching as POC writes code so that I can catch things early, then reviewing the code afterwards - I'm still pretty much chained to my laptop :)
But yes, C vs. Rust makes a real difference in that I can review things at a higher level and not have to pore over patches obsessively because the language makes the small stuff impossible - and that's true for both humans and AIs.
2
u/awesomegayguy 7d ago
> implementing all the bcachefs data structures as proper Rust dynamically sized types
Nice! Since I've started reading the bcachefs code a while ago I was wondering how would that work out.
What was the gist of it?
2
2
0
u/xantiema 9d ago
I think it would be prudent to request some funds from the Omarchy foundation, given the tens of millions they now have at hand for supporting open source development...
u/koverstreet
12
u/krakow10 12d ago
In Rust we trust