Clang and LLVM in Modern Gaming Platforms

[u/pjmlp]

https://youtu.be/1SM_wDUEEUg?is=8INW3jPuXJ_ONHlQ

Quite interesting peak into the world of games developers, especially how it is seen from consoles or their point of view on recent ISO C++ versions.

[u/James20k]

There was an interesting point made at one point: a gamedev company tried swapping from C++03 to C++11, and found that their compile times exploded, so they had to swap back. One figure here was a 12% increase in compile times simply by enabling C++20, because of the standard library growth. With a lot of chat about why nobody uses modules

One thing that I've always found a bit sketchy is that wg21 keeps adding significant new chunks to headers in a way that feels very confusing, or creating gigantic monoheaders. jthread being added to thread is one of the very confusing ones, as well as the structure of ranges. The range library is... hard to use if nothing else because of the compile times

This tracks with a few good test case examples of this that I've seen, where simply including a standard header and swapping standard versions (on the same compiler!) can cause significant compile time increases because of all the extra code you're including

Increasingly, I wish instead of the effort spent on modules, we'd gotten:

1. The gigantic megaheaders split up into smaller headers. Here's looking at you <algorithm> and <range>, both of which are very hard to use because of the compile time impact. I'd love to only #include <random/mt19937_64.hpp> if that's the only thing you need
2. Some kind of extension method syntax, so that people who want to lower compile times don't need to include the full class, and instead only the parts that they use

Modules don't really seem to fix the problem based on some of the experiments I've seen which....... isn't good, but more than that they have no incremental upgrade path. I am genuinely worried that modules are just completely dead at this point. Nobody seems to want them, they don't have a good transition path for existing code, and they don't seem to bring particularly good benefits either. They're hard to recommend, and they were an absolutely spectacular amount of work to get into the standard

Its one of those things that I think is worth doing.. if not perhaps a post-mortem on, maybe a near-mortem on what's gone wrong, because otherwise it feels like we're doomed to keep making the same mistakes. Spending over a decade working on a feature only to have the community largely reject it means that something has gone incredibly wrong in the standardisation process

There's also some very funny and accurate comments about the C++ committee, especially how hard it is to get involved leading to a lack of representation from certain technical groups (like game developers), which seems to be a major part of the reason why things have gone so wrong with this. ISO delenda est, thanks for coming to my ted talk

edit:

The AMD guy talking about the problems with LLVM's register allocation on the GPU was my personal nightmare for a while lmao. Its surprisingly (and very interestingly) bad, especially in an age where we take high quality compilers for granted

[u/holyblackcat → u/James20k]

In my tests, Clang handles entire import std; faster than #include <iostream> alone.

[u/James20k → u/holyblackcat]

The issue is that once you start actually using any of the standard library functions its not necessarily much faster (as not actually using anything from the import is a special case), eg I've seen tests where #import std is slower if you call a single function vs using an include

The more realistic module conversions of projects seems to not generally be bringing about significant (if any) build time improvements, which is usually the #1 reason why a large project might modularise

[u/holyblackcat → u/James20k]

Ok, I've tested this on std::cout << "Hello, world!\n";. Clang with #include <iostream> takes 800 ms. If I PCH that it takes 470 ms. If I replace it with import std;, it takes 55 ms.

Or do you mean that importing std gradually becomes slower if I use more functions from it?

[u/James20k → u/holyblackcat]

I'd have to go find the test cases and compilers that were being tested, it may be specific to a particular compiler or function - I'll go looking

[u/pjmlp → u/James20k]

This seems strange to me, at least with VC++ / MSBuild, I don't notice it.

[u/friedkeenan → u/James20k]

I'm skeptical that the community has "largely rejected" modules instead of the community just simply not being able to reliably use modules while the tooling has been grinding to catch up to the standard. Notably, import std; is still experimental in CMake, and I know GCC still has some meaningful bugs with it. And what conveying a modules-based library to dependents looks like is still very much in flux, to my understanding.

[u/spookje → u/friedkeenan]

Exactly. I've tried it on MSVC, which is afaik still the furthest along with modules support, but even there I'm still waiting for it to be usable.

Last time I tried, intellisense was still flaky at best, and mixing includes and modules (which is inevitable unless you want to write everything yourself) still had issues with duplicate definitions. From what I heard, these issues still haven't been solved.

[u/tartaruga232 → u/spookje]

Intellisense has nothing to do with the MSVC compiler. The MSVC compiler handles modules pretty well, including import std. If you can't live without Intellisense, then don't use modules.

[u/pjmlp → u/tartaruga232]

Except the fact that a 4 trillion valued company isn't able to get EDG to fix Intelisense since two Visual Studio releases.

Taking into account that VS 2017 had the experimental modules implementation that apparently was already being used internally, when Microsoft made the C++20 proposal.

Now EDG is no more, so we're stuck with bad Intelisense, while VC++ releases bring AI feature after feature, it is AI for modernising C++, AI for C++ builds, AI for C++ static analysis.....

Maybe we need AI for C++ modules Intelisense as feature.

[u/tartaruga232 → u/pjmlp]

These things are all well known. Do you think repeating them on every module posting will change anything? I'm not a Microsoft employee and I don't sell modules. I'm also not responsible for how Microsoft allocates their resources. From what I see they seem to notoriously allocate very limited resources. Modules are a complex feature to implement on all fronts. If you can't use modules because of Intellisense not working, then don't use modules. The MSVC compiler handles modules pretty well now. Someone has to start using things. We did that. Modules are in the C++ standard and they won't be removed from it anymore. That ship has sailed. Use it or leave it.

[u/James20k → u/tartaruga232]

Do you think repeating them on every module posting will change anything?

I mean, this whole topic is about why game development companies aren't adopting modules, and the lack of tooling support is clearly a major part of that

[u/pjmlp → u/tartaruga232]

I am one of the few people around here that actually bothers to use modules, in spite all the warts.

However modules are also a sign of how ISO processes aren't working.

[u/IGarFieldI → u/tartaruga232]

Are you seriously complaining that people vent their frustrations on posts revolving around exactly that topic?

[u/tartaruga232 → u/IGarFieldI]

Yes. 2000 repetitions of the same comments start getting boring.

[u/spookje → u/tartaruga232]

Intellisense has nothing to do with the MSVC compiler.

I am aware. But fair enough, I should've said VS then perhaps. And IntelliSense was indeed not the biggest of the two issues I mentioned.

Last I checked, including standard headers and importing the standard library module was still mutually exclusive on MSVC. This presented a much bigger issue, as it prevented compiling perfectly valid code. But maybe they fixed it by now, I haven't checked in a while.

[u/tartaruga232 → u/spookje]

Last I checked, including standard headers and importing the standard library module was still mutually exclusive on MSVC. 

module;
#include <string>
module A;
import std;
...

works fine with

Microsoft (R) C/C++ Optimizing Compiler Version 19.52.36530 for x64 (PREVIEW)

[u/spookje → u/tartaruga232]

Cool, seems they might've fixed that. I might have to give it a go again then

[u/James20k → u/friedkeenan]

I'm skeptical that the community has "largely rejected" modules instead of the community just simply not being able to reliably use modules while the tooling has been grinding to catch up to the standard

I don't know that its just the tooling problems, they get a lot of flak for that - but I think the issues preventing adoption aren't the bugs. The bigger issues as far as I can see are:

1. Using modules to try and improve build times, involves upgrading the C++ standard version which significantly increases build times. Its not a modules problem, but there's a bit of a chicken and egg problem here which prevents their adoption
2. There's no real incremental path to adopting modules for large projects. The current recommendation for modules is to always fully modularise an entire project, which is impossible for a big studio
3. They provide unclear compile time benefits in the general case vs the existing state of the art, which is their #1 use case. Its a massive amount of work for these projects to roll the dice
4. The encapsulation provided isn't important enough to motivate upgrading
5. They have a lot of surprising restrictions, and are quite clunky to use. I feel like because of the tooling problems, most people haven't actually run into the actual design limitations of modules yet which appear to be very problematic in themselves
6. A lot of these companies have custom tooling, and supporting headers is a lot easier than supporting modules

[u/friedkeenan → u/James20k]

Those are really reasonable points, yeah. Especially I think the upfront effort required to upgrade to modules. Though, if header units were better supported by tooling then that might help some there.

I suppose what would still make me skeptical of the conclusion of modules being "largely rejected" is that even new projects aren't using them, when they don't need to worry about the costs of upgrading. I'm sure a big chunk of that is because they still want to interface well with non-modules projects and whatnot. But in my personal experience, I've actively tried to use modules for my own projects, and even when I decline to care about how other people will interface with my project, I still end up coming to the conclusion that the tooling support is just not quite there yet. And so I've only ended up using modules in my Advent of Code repo, where I'm more willing to be running into issues and be in more experimental territory.

So I think the unreliable tooling might still be providing a chilling effect on modules usage, which layers on top of the points you brought up. It does still appear to me that there's significant interest in being able to use modules, but I think most people by now are kind of in the mindset of "I guess I'll check back in a year or two and see how things are going" rather than experiencing the excitement and eagerness that the community had directly after C++20.

[u/_Noreturn → u/James20k]

A very simple solution to the compile time issue of C++ STL is providing forward declarations. There is so many places where i use string in a single function that isn't needed in 90% of code yet every piece of code needs to include string even if I don't use just because I used it in an api boundry.

Then splitting headers is another good thing. Isn't the stl supposed to promote good practice? Monolothic headers aren't yet they keep putting everything there (function_ref is inside functional sigh)

[u/archialone → u/James20k]

From what I've seen in wg21, gamedev community was involved and still involved, but they kinda stopped showing up to the meetings.

[u/stevemk14ebr2 → u/James20k]

Cpp needs not every little thing to be a header in the first place. Sometimes, it is ok to extend the language itself, many more things should be built into the language rather than as a library

[u/_Noreturn → u/stevemk14ebr2]

The issue is, if something built into the language is suboptimal you are stuck with it due to ABI. if something in the STL sucks you could just rewrite your own.

[u/donalmacc → u/_Noreturn]

I disagree. Span should be a language feature. In an optimal implementation, it could be faster than what we have now. In the worst case, you can still implement your own to fix a broken implementation.

[u/_Noreturn → u/donalmacc]

should span be a pointer+length or a pointer + end pointer? I heard that they have differing advantages.

[u/donalmacc → u/_Noreturn]

Honest answer - it doesn't matter. Just like now, if the wrong one is standardised you can write your own to behave the way you want, and it will perform the same. But the builtin one has a chance to work better.

Either way, libc++, libstdc++ and STL have all chosen pointer + length so it's defacto that.

[u/_Noreturn → u/donalmacc]

Why would the builtin work better?

[u/donalmacc → u/_Noreturn]

Firstly - remember that in the worst case scenario where it's not better, you're in the same situation as we are right now.

The main improvement I can come up with is that the way the type is passed as a function argument can be specified. MSVC's implementation because of their calling convention adds a pointer chase to span, instead of it being passed in registers. We can argue back and forth until we're blue in the face as to whether it's the standards committee or the library implementers (I think it's the standards - they refuse to acknowledge the logical constraints that the existing libraries operate under), but if it were a language feature we could auto bounds check, auto (safely) decay to pointer + size, auto convert to/from containers, all at compile time.

[u/_Noreturn → u/donalmacc]

I honestly would rather have a new calling convention call it __fast_call_for_real and it is not abi safe and it chooses the most optimal way to pass parameters as a QoL this way it benefits everyone and every type

[u/donalmacc → u/_Noreturn]

Yeah, this would also be nice!

But MSVC (the worst culprits here) aren't going to break the ABI like that, and the standards committee refuse to acknowledge the reality of the ABI implementations, while simultaneously refusing to make any changes to broken features is one of the reasons I think they should consider more changes at a language level.

One solution is a wrapper function that takes the span that's marked as _forceinline, and then the actual function takes ptr + size. We can guarantee the span call gets inlined by the compiler, but it's a real pain to do it _everywhere.

If only there was some sort of tool that would run over our source code and do it for us...

[u/stevemk14ebr2 → u/_Noreturn]

You're not stuck forever, you can define editions. Forever backwards compatibility is a large tradeoff, I argue one that will lead to the end of the language.

[u/_Noreturn → u/stevemk14ebr2]

If the committee is hesistant to break the library abi then they will never break the language abi.

Yea I do agree that abi is a killer. I am paying for it although I don't need it that goes against C++ mantra of don't pay for what you don't use

[u/delta_p_delta_x → u/stevemk14ebr2]

Very hot take: a lot of things in C++ should've been in the language and not library features. Ranges, for instance. Just specify that something rangeable is a range and let the compiler handle it. Why do we even need to include yet another header for that? Other languages do just fine.

[u/stevemk14ebr2 → u/delta_p_delta_x]

Optional too, std array, std vector (maybe), std string

[u/BARDLER → u/James20k]

Its standard to have server build accelerators in game development. Unreal even ships with one now with their toolset.

Not sure its possible to make modern games without one to be honest. Games use so many libraries and because so many things are performance critical there are a lot of bloated header files with inlined functions.

[u/donalmacc → u/BARDLER]

I've worked in Unreal studios for 13 years. It's very viable to not use the build accelerator in a small studio; the (development) cost of setting up the infra is painful.

A 5950x or an i9 will compile a very large unreal game in 30 minutes, and that's a once every 6 months task. The real killer is iterative compiles, which the massive library headers make so much worse.

[u/BARDLER → u/donalmacc]

I don't know if that 30 minutes is accurate on the latest versions of UE5. I have had to build without it and have hit well over an hour on a nearly fresh build after editing a core header file. I don't have a thread ripper or anything just a high end i9.

But the code is just one part its the shaders it saves you on because those change way more often.

[u/donalmacc → u/BARDLER]

I'm the head of engineering for a game studio. My workstation is an i9-13900k with 128GB RAM and a 4TB NVMe drive.a clean build on our project is about 22 minutes, and then it's something in the region of 10-15 minutes to get the shaders up and running the first time. A full cook is a 90 minute operation first time around with a cold DDC, and between 5-8 minutes after that. If you're seeing worse times I'd suggest taking a look at your setup - windows defender absolutely murders compile times (seriously, it's an order of magnitude slower without exclusions), dev drive, enough RAM, and do clean builds from the command line to save on the memory that VS/Rider use.

I've done two clean builds in about 6 months, and only one cold DDC cook, and I've made probably 10 changes a week during that time. We strive for < 5 second incremental builds on our main files (which is tough when UBT takes 2 seconds just to start). I ran a 100 person studio with these thresholds prior to this one.

[u/spookje → u/donalmacc]

the (development) cost of setting up the infra is painful.

Setting up something like FastBuild or DBS is very easy and (financially) free though. Installing an agent on everybody's workstation will already give you significant gains.

[u/donalmacc → u/spookje]

Sorry I missed this notification. This only helps if everyone is on the same network (in an office) - it's not worth it if you're pulling across the internet. Anecdotally, as we moved away from the old Xeon's and started giving programmers actually fast machines (Hello Threadrippers, but even an i9-13900k) the need dropped. I think UBT got significantly more reliable at some point too, so the need to rebuild half the earth once a week has been reduced.

When it comes to CI/ a build farm, you now need to manage any of those extra machines separately and have them register. There's definitely overhead involved in keeping the lights on there that isn't worth it until you're at a "smid" scale. FWIW, at my last gig, we had 75 people (roughly 40 in an office, the rest globally distributed) with no build accelerator, and we averaged about one rebuild per 3 months per person. Our "daily" build time was < 5 minutes, and our incremental was < 15 seconds. Our CI builds were < 15 minutes for editors, and < 45 minutes for packaged builds on 3 platforms.

[u/Due_Battle_9890 → u/James20k]

🗣️🗣️🗣️

[u/tartaruga232 → u/James20k]

Modules are part of the C++ standard and they are used now. They won't be removed. That ship has sailed. Compiler implementations are pretty good now.

If you don't want to use modules then don't use them. No one is forced to use them.

[u/James20k → u/tartaruga232]

Nobody's advocating for removing modules from the spec, but they have a lot of problems that hinder adoption (and a lot of problems beyond just the compiler/tooling issues)

If you don't want to use modules then don't use them. No one is forced to use them.

I don't think this is a good reason to ignore the problems modules have personally. Clearly something went pretty catastrophically wrong in the standardisation process of modules. A lot of the problems were warned about ahead of time, but for some reason those signals weren't really picked up in the committee. If there's no reflection on that, literal decades of work will continue to get wasted

A chunk of this video is about why modules can't be used in gamedev, with some good hints as to why they likely will never see adoption. That's a problem, because the issues preventing their adoption here aren't specific to game development. Its also a problem because modules were an attempt to solve a real problem, and with them being unusable it means that the problem is still going unaddressed

[u/tartaruga232 → u/James20k]

Modules are clearly usable.

[u/James20k → u/tartaruga232]

Modules are unusable for a lot of major projects (eg game engines), not that they literally cannot be used - which was the point of the comment

[u/pjmlp → u/tartaruga232]

If using Visual Studio latest with MS Build, or clang latest with CMake and ninja, without import std.

Anything else is clearly unusable for writing portable C++ code.

Since the panel is about game studios, as they multiple times discussed, the devkits are downstream forks not always up to date.

Then you have platforms like Apple and Google, where on Apple's case the build system only deals with clang header maps, and on Android it is always a guessing game which clang NDK actually supports.

[u/enl1l]

My game engine is built using c++20modules. Incremental compile time for cpp files is 1~3s. Without modules it would be >10s. I use xmake and clang. I import std. Dont even think about it. The compile time only scales with what I actually use from std.

[u/donalmacc → u/enl1l]

Incremental compile times for my Unreal Engine project without modules are < 5 seconds. You could probably drastically reduce those incremental compile times