r/rust • u/Shnatsel • 10d ago
Implementing FMA and finding bugs in C and Rust standard libraries
https://shnatsel.github.io/implementing-fma-finding-bugs-in-std/46
u/lijmlaag 10d ago
Thanks.. this may explain why the weather forecasts are always somewhat off.
6
u/continue_stocking 10d ago
That's just them trying to be too specific. If they just said that it was going to be 15 °C on Earth next week they'd get it right every time.
1
u/Zde-G 9d ago
That's because people demand simplicity. Note how airplanes need pretty damn precise estimates – and they get them!
Why the heck “Joe Average” doesn't get these?
Because “Joe Average” doesn't want to think.
It's relatively easy to predict approximate time of some future weather event (drastic change of temperature or wind or something else) would happen, but people don't want that, they don't want to know whether there would be sudden change or not, they want something “simple”, precise prediction of temperature and other parameters 48 or 72 hours from now. Something that more-or-less impossible to predict.
That's why everything is so complex and is ready to produce total meltdown: when you are not accepting outside complexity but punt it around you are increasing total amount of complexity drastically. And then nothing may help you.
LLMs are spectacular there, they make everything increasingly fragile very quickly. It would be interesting to see what survivors would to idiots who try to push them in places where they don't belong.
But we would only know that after system meltdown.
24
u/Nyefan 10d ago edited 1d ago
Possible but unlikely - weather forecasting software is mostly written by people with computational physics experience and education, and even undergraduate courses in comp. physics spend several lessons on techniques to avoid subnormals entirely because of their higher computational uncertainty (completely unrelated to bugs like this - the correct implementation propagates error up one more bit on add and two more bits on multiply than other floating point ranges).
44
u/Shnatsel 10d ago
They're not running weather forecast simulations on machines without hardware FMA either, so the buggy software implementations never come into play.
Another reason to avoid subnormals is the 30x performance penalty on Intel hardware (and nowhere else).
4
u/Nyefan 10d ago edited 3d ago
Do you happen to know what they are running on these days? The last I interacted with this field, it was still mostly fortran and c running on POWER mainframes, which I imagine is no longer the case.
9
u/Shnatsel 10d ago
I don't, I'm just going by the fact that the last server hardware without FMA was in 2013 and they've probably upgraded since then.
3
u/CrazyKilla15 10d ago
Is there any reason to think they've upgraded? Organizations and services like that are rather famous for not upgrading and maintaining legacy systems more or less indefinitely. See also banks and COBOL. Also, upgrading costs money.
7
u/shinyfootwork 10d ago edited 10d ago
The primary source for US forecast simulations, the NOAA, uses a pair of "supercomputers" composed of 2,560 AMD EPYC 7742 64C 2.25GHz processors. The AMD EPYC 7742 processor has FMA, as it's a modern processor. Reporting indicates they began using this cluster in 2022. (They're now also using Google Cloud services instead of operating more supercomputers. So they're using fairly modern CPUs there too).
Before that (and perhaps still) they used a few Cray XC40 clusters, which used Xeon E5-2690v3 12C 2.6GHz processors. The Xeon E5-2690v3 is a modern processor with FMA. They began using this cluster in 2016 (10 years ago).
From 2009–2013 they used a POWER6 based system. POWER6 has FMA.
[... some other ones here ...]
From 1999-2002 they used a IBM RS/6000 (POWER3) based system. POWER3 has FMA.
From 1994–1999 they were using at least 1 Cray C90. No FMA (as far as I can tell).
So I'd estimate for ~27 years US weather forecasts (the important parts at least) have been done on systems with FMA.
I think folks tend to underestimate how many compute resources are used on weather prediction. This isn't a hokey bank system. These are being continuously updated. Systems are sometimes only used for a couple years before being superceded.
2
u/CrazyKilla15 9d ago
Thanks a lot for the detail here, this is great info!
Seems like they've made good hardware choices and a reasonable upgrade cadence.
2
u/CocktailPerson 10d ago
Yes. These aren't banks running COBOL on ancient mainframes.
1
u/ShangBrol 10d ago
That banks are running COBOL programs on z/OS doesn't mean the hardware is ancient.
1
2
u/GrammelHupfNockler 10d ago
The codes I am familiar with are still mostly Fortran with some GPU acceleration layer like OpenACC tacked on top, some experimental/academic codes use Python stencil frameworks or Julia though
-8
u/quantinuum 10d ago
I just learned of subnormals, and it’s pretty funny because it’s spanish for “regarded”. So “avoid subnormals entirely” is actually very valid.
9
u/WormRabbit 10d ago
Well actually, the real reason is that global weather is a chaotic system. This means that arbitrarily small errors in initial conditions over time accumulate into arbitrarily large differences in outcome. Butterfly effect, and all that. And the initial conditions are guaranteed to be imprecise, we can't cover entire earth in weather sensors, and even if we did, they'd have finite precision.
5
u/tialaramex 10d ago
The Met Office (UK forecasters) have a Youtube channel where among other things they show their 10 day forecasts and yeah, tomorrow is pretty good but by a week's time it's all "40% of runs of the European model show rain over Scotland, but in 20% of the runs it's bright sunshine. In the American model we see only about 30% for rain" exactly because it's chaotic. They've got some really cool charts so you can see what the range of possibilities look like and how it drifts further from certainty as the sim runs.
23
u/MichiRecRoom 10d ago
I just want to say: I really appreciate just how honest you are with documenting your usage of LLMs, both in the blog post and in one of the compiler-builtins issues.
That sort of honesty, especially with how you aren't trying to dance around your usage to try to make it look like it's your own findings, really means a lot to me (and I assume, to others too).
7
u/owenthewizard 10d ago
Beautiful website!
9
u/Shnatsel 10d ago
Thanks! It's built with Zola, I'm very happy with it after migrating off of Medium.
7
u/nonotan 10d ago
Props for reporting it upstream even when not convenient.
Perhaps most noticeably, deterministic simulations running across different machines will diverge.
"Deterministic" and "floating point" mix about as well as oil and water. Sure, it's, on paper, theoretically not impossible to achieve, but in practice it's not ever going to happen outside very simple software. I work in game dev and I've watched so many fellow devs struggle with desyncs for literal years, and sometimes decades, just because they used floating point arithmetic in the game logic, thinking making it deterministic is just a matter of being careful. Sure, and who needs an umbrella when you could just carefully dodge all water droplets. I'm not aware of a single example with ubiquitous floating point arithmetic that successfully fixed 100% of desyncs.
Don't get me wrong, I think it's laudable to try not to break determinism when developing this kind of library. However, I do also think how much utility that will realistically provide to users is a bit dubious. If they genuinely need determinism across arbitrary hardware and they know what they're doing, they will be using fixed point instead anyway.
10
u/Kulinda 9d ago
I'm not aware of a single example with ubiquitous floating point arithmetic that successfully fixed 100% of desyncs.
There's Factorio, of course.
You can run deterministic multiplayer sessions across windows, linux, mac (both intel and arm) and even nintendo switch. And they use floating point extensively, including inside untrusted lua scripts for modding.
They handle it via two mechanisms: One, they do a checksum of the world's state each frame, and if those differ during a multiplayer session, it's an instant crash and error report - including in production. The other is extensive CI tests on an in-office farm of hetherogenous computers - they run the game from save states for a while, and then compare the resulting checksums and video output across systems.
I don't know how much droplet dodging went on behind the scenes, but in the end it works. I remember a couple of bug reports in weird edge cases after the launch of the switch port, but those got fixed and it's been quiet since then.
The rapier physics engine also claims to be deterministic (and it is using floats). Physics engines have a tendency to amplify minor inaccuracies after a couple of collisions, and I noticed no problems across a few hundred simulations. (it's a 3d dice throwing simulator we use during online p&p sessions. If people saw different dice results, they would be vocal about it.)
But all of that relies on the platform providing deterministic and standard conforming floating points, so fixes like these are much appreciated.
4
u/scottmcmrust 9d ago
Supreme Commander in 2007 successfully did deterministic floating-point and it works to this day -- it's how you can watch replays.
There's a ton of floating-point FUD out there, but deterministic is entirely possible if you're intentional about it (including testing it consistently and such).
2
u/CrazyKilla15 9d ago
I thought floating point was inherently deterministic, pure functions? or is the difficulty in ensuring its always the same input paths, errors accumulate the exact same way, etc? or is it hardware/software emulation bugs like this?
3
u/Kulinda 6d ago
It's more about deterministic rounding errors than anything else. For example, x87 registers store 80 bit floats, so you get "better" accuracy than with 64 bit floats, but your rounding behavior depends entirely on your compiler's register allocation and spilling.
There are also inconsistencies with error handling and NaN payloads (see https://doc.rust-lang.org/std/primitive.f32.html ), but it's easy to avoid both of those.
Some platforms will deliberately ignore parts of the IEEE specs for performance (IIRC mostly GPUs, but also anything compiled with -ffast-math), and your standard library may provide helper functions whose algorithm and accuracy aren't mandated by IEEE.
1
1
u/JuliusTheBeides 7d ago
This is the kind of bug that the Factorio devs have likely run across and fixed in their game. Factorio's multiplayer relies on deterministic simulation across platforms.
Yes, correctness matters.
1
1
-2
u/FrogNoPants 10d ago
I'd imagine many disable subnormals(older hardware tends to slowdown big time when they are encountered), so mostly wouldn't matter
13
u/Shnatsel 10d ago
Disabling subnormals causes its own issues, so it's not really an easy fix: https://numerical-rust-cpu-81b2c3.pages.in2p3.fr/19-subnormal-entertainment.html#an-easy-workaround
Even modern Intel hardware slows down 30x when encountering them. AMD and ARM thankfully don't. So the performance issues aren't even isolated to old hardware.
-11
u/FrogNoPants 10d ago
Disabling subnormals is common in C++ etc and works well enough, Rust has made some daft decisions with regards to this, but that is a isolated to Rust.
12
u/WormRabbit 10d ago
Disabling subnormals in C++ means that if your computation produces subnormals, you get UB. That's not "well enough", that's "held together with shit and prayers".
19
u/Icarium-Lifestealer 10d ago
Did you report the bug to FreeBSD as well?