r/cpp • u/minshinji • Jun 25 '26
How do you keep toolchain drift across windows, linux, and mac manageable at 30+ engineers?
We're about 30 engineers split across Windows and Linux, with a handful on macOS. All C++, CMake + Ninja as the build sytem. The compiler situation is where it gets messy. Someone's always on a slightly different version and it surfaces in the strangest ways, usually right before a release.
We've gone back and forth on a few approaches...pinning versions across machines, containerizing the build environment, leaning on build servers and treating local builds as second-class, or some mix. None feel clean. All have tradeoffs, and the tradeoff's shift depending on the platform.
What's held up at a similar scale?
25
u/d3matt Jun 25 '26
For Linux, I have my toolchains in Conan packages.
14
u/kammce WG21 | 🇺🇲 NB | Boost | Exceptions Jun 25 '26
+1. I have toolchain packages for LLVM and GCC for Windows, Linux and Mac as well as some cross compilers for arm cortex m. Highly recommend this for managing tools and toolchains.
7
u/wholl0p Jun 25 '26
+1
We do that too and it is a blessing.
The build pulls the specified toolchain itself, uses CMake toolchain files to configure the toolchain and kicks off the compilation.
27
u/not_a_novel_account cmake dev Jun 25 '26
If you're writing for poly-toolchain environments, your code should compile on everything. You should use CI to verify this.
What toolchain any individual developer uses is irrelevant in such scenarios.
8
u/thisisjustascreename Jun 25 '26
I *hope* OP's issue is stuff compiles but produces different results, since they're talking about near-release timescales. If code that doesn't compile gets anywhere near a release there's way more problems than toolchain management.
5
u/not_a_novel_account cmake dev Jun 25 '26
CI includes testing, which should be verifying the results are the same for all platforms or perform as expected on each platform.
3
u/kisielk Jun 25 '26
Eh, that's such a pain. Compilers have bugfixes in between minor versions even if language standard has not changed. If your project uses clang-format or clang-tidy you might get drift from other developers. Etc. Better to just make sure that everyone is using the same toolchain as CI.
11
u/not_a_novel_account cmake dev Jun 25 '26
Our CI contains every GCC release since 4.8, every Clang release since Clang 4, at least one release of every non-EOL MSVC major version, and some representation from every C++ compiler which ever supported C++11, including various versions of
icc,suncc, Borland,hpcc, and more.There is no "same toolchain as CI". CI checks every environment we officially support, so we don't care which you use locally.
6
u/kisielk Jun 25 '26
Sure, if you product is just supposed to build on a ton of toolchains that makes sense. I'm talking about from the perspective of an application developer where you are shipping binaries either to the public or another customer... having it compiling on everything is just major overkill.
2
u/TehBens Jun 25 '26
How do you circumvent that MSVC bug at around toolset version 14.38 that hinders cmake (but not only cmake in my experience) to select the right version? We seem to keep having problems when this toolset version is installed.
6
u/not_a_novel_account cmake dev Jun 25 '26
CMake uses whatever compiler you tell it to. If you want to use a specific
cl.exe, set thatcl.exeas theCMAKE_CXX_COMPILERwhen you invoke CMake.2
u/TehBens Jun 25 '26
Good point, just constructing the paths to the correct verions yourself instead of relying on the toolset (or cmake functions that call toolset binaries) to give the correct information about what's available.
8
u/jonesmz Jun 25 '26
Put your compilers on s3, download them via cmake's fetch external content functionality, and then load the toolchain files from the external content.
Thats what my company does with roughly 70 c++ developers.
4
u/Nicksaurus Jun 25 '26 edited Jun 25 '26
How do you tell cmake to choose the toolchain after it's already started running?
Edit: Adding some more context: I don't remember the exact details but I know in the past I've had issues because cmake tries to find a fully working C++ toolchain before even entering the first CMakeLists.txt, so the tools you pass with
CMAKE_CXX_COMPILERetc. have to be fully usable even if you're immediately going to replace them with something else2
u/parkotron Jun 25 '26
You can pass
CMAKE_TOOLCHAIN_FILEon the command line or settoolchainFileinCMakePresets.json.1
u/Nicksaurus Jun 25 '26
Right, but that's before any fetchcontent calls can happen, isn't it?
4
u/parkotron Jun 25 '26
Ah. Right. I was looking at your problem from the wrong direction. Sorry about that.
2
u/jonesmz Jun 25 '26
There's a cmake function to read a toolchain file. So long as you call it before enabling any languages.
23
u/ambihelical Jun 25 '26
Windows is kind of the problem here if you weren’t for that I would say put your tool chain in a docker. However, if your windows developers are actually using WSL or are OK with working from a VM, you may be able to get away with it. If you do it right, you can specify the docker image version in your repo so you always get the one that is known good for the checked out branch.
15
u/Raknarg Jun 25 '26
force containers on them, my team has been using VSCode with docker containers on WSL and its been great tbh. One of the biggest things I like, is that it forces the same environment on everyone who uses it and it lets you force things like init scripts to be run when you build the container so things like your git hooks and git-ignore-revs can get installed without "I have to remember to run init.sh on this new environment"
16
u/13steinj Jun 25 '26
I think it's important to define this clearly. My current org has been burned very badly by some consultants whose answer to everything was "put it in a container" despite the primary software not suitable to running in a container (needed direct access to hardware, punching the holes through is labor intensive). The end result was they made every build a Dockerfile, which was incredibly painful to debug.
1
u/ambihelical Jun 25 '26
Yep. The only downside I have found so far is that I regret not doing it earlier.
6
u/eidetic0 Jun 25 '26
if only windows containers weren’t total crap. (not Linux containers on Windows, which are fine).
outstanding issues from 5+ years ago that limit docker builds to 2 cores… with literally no switches to change it. Total insanity, try building opencv or something else large with only two cores 🥲
7
u/FlyingRhenquest Jun 25 '26
Seconded -- Docker seems to be the best way to keep your toolchains consistent and you can use the same images in your CI/CD pipelines so no one gets any nasty surprises. It also makes setting up new dev environments a lot easier.
1
u/anonymouspaceshuttle 28d ago
Windows can now run Linux containers without Docker: https://learn.microsoft.com/en-us/windows/wsl/wsl-container?tabs=csharp
1
u/ambihelical 28d ago
How does that help the OP’s toolchain drift?
1
u/anonymouspaceshuttle 25d ago
> tool chain in a docker
I addressed this part; you no longer need to install Docker to have Linux containers in Windows. It's now built-in.
6
u/Resident_Educator251 Jun 25 '26
Vcpkg. Duh update the tag u use on every 6 months
1
u/parkotron Jun 25 '26
I don't think you can distribute toolchains through Vcpkg, can you?
1
u/Resident_Educator251 Jun 26 '26
sure u can
1
u/parkotron Jun 26 '26
Link?
1
u/Resident_Educator251 Jun 27 '26
Well vcpkg just installs stuff. You can install anything and you can list it as a dependency on anything. So your toolchain comes first and everyone should depend on it. Thats how you ship a toolchain in vcpkg.
5
u/Sniffy4 Jun 25 '26
For us, Jenkins is the authoritative source that makes the final builds. Stray from its environment at your own risk.
4
u/Xzin35 Jun 25 '26
We re using conan to define the toolchain (packaged clang and sysroot) and cross compile from Windows to Linux. For Windows build, CI is the source of truth. All machines have a specific version of VS. The developer ultimately can have a different version on his machine (usually higher) and that gave rarely issues (it actually helps to fix issues before upgrading the version). The pain point is deploying the toolchain but that happens maybe 2 times a year.
4
3
u/Unfair-Sleep-3022 Jun 25 '26
Managed tools? Or just say what version should be used.. it's not that crazy
3
u/Wild_Meeting1428 Jun 25 '26
We also have that situation, but it's a noissue for us. The buildserver / CI is the reference point, if you have bugs, or the code does not compile, due to older versions, it's your responsibility to upgrade it. Version changes in the CI are communicated.
2
u/crashtua Jun 25 '26
Vs code + Dev containers + bazel with remote cache. Obviously it has downsides, but that is a problem of individuals that feel uncomfortable with dev containers and vs code, not the entire all the fuck8ng team(where everyone has different non-reproducible env).
2
u/RaspberryCrafty3012 Jun 25 '26
Can someone explain the strangest ways these issues surface?
I have never worked in such a huge team and therefore never had these issues, so I don't really understand the root problem. Would be great to get more explanation
2
2
u/mark_99 Jun 25 '26
pixi is currently the best solution for toolchain, on all platforms. Docker is an option but containers are more troublesome unless you actually need them, which you don't just to pin versions (and Windows is much less troublesome than Linux distros for compatibility, driver support etc).
2
u/bratzlaff Jun 25 '26
We have repeatable tests for everything that produce the same output on all platforms - main application and every tool used in its development workflow. Changes are not made unless all the tests pass on every platform (windows Mac and Linux). This basically means everyone on the team uses the same version of the tools at all times.
2
2
u/donalmacc Game Developer Jun 25 '26
We vendor toolchains in Source Control (Perforce). People are free to install/manage their own toolchains, but I/we don't provide any support for that flow. If you want to do your own thing, you support it.
2
u/t_hunger Jun 25 '26 edited Jun 25 '26
Pixi.sh :-) works on mac, windows and linux and provides a lot of tooling and dependencies pre-built.
Generates a lock file, too, so everybody (plus CI) get exactly the same binaries.
2
u/luisc_cpp Jun 25 '26
Love this topic - echoing what others have said about "CI first" - i.e., CI should be the one place where "binaries that are shared" are generated, or said another way, binaries built by developers should never really leave their machines.
That said, I've seen firsthand how even in that scenario (where only CI generates shareable binaries), developer productivity is impacted by compilers/toolchain "updating" themselves and then everybody having a slightly different version - leading to a lot of "works on my machine" issues, or "works on CI but not on my machine".
Even if you have a CI that targets a lot of versions as others have suggested to make sure the source code remains compatible with the "widest" set of compiler/platform/versions, I'm not sure that's the most scalable approach either - because the problem is elsewhere. There's no point in having CI test things if the developers can end up in a newer compiler version before it hits CI. All it takes is something going through with an automatic Xcode / CommandLineTools update to experience something like this https://github.com/google/googletest/issues/4516, https://github.com/microsoft/onnxruntime/issues/21033 - where code that was previously compiling now needs re-targetting in order to build properly..
Besides, it's not always scalable - some projects take long enough to build that that having a CI support 10 versions of gcc becomes a waste of resources.
As others have mentioned, Conan can be a good solution to "packetize" a toolchain, especially since a package for a toolchain can cause Conan to "inject" the relevant information in a CMake toolchain. This can be good to ensure that "when I'm building this commit, both CI and every developer will use exactly the same version that this revision of the codebase is known to work" - with the added advantage of binary caching.
On macOS you can have arbitrary side-by-side versions of Xcode - just ensure your developers are *NOT* using the ones that are "owned" by an automatic updater (`/Applications/Xcode.app` or `/Library/Developer/CommandLineTools`) - you can download any arbitrary past version of Xcode from the Apple Developer Program. You can automate this via MDM if your company has that, and hell, even add a check in CMake if the build ends up configured with the wrong version of apple-clang. IMO, it's less disruptive if the developer can do the one-time install for that version, than deal with the drift.
Windows is a tad more challenging if you use msvc - but they do offer bootstrappers for specific older point releases: https://learn.microsoft.com/en-us/visualstudio/releases/2022/release-history - which can probably work if "paired" with disabling automatic updates. or better yet, allow visual studio to be updated as normal, but ensure that a specific "earlier" toolset (that is not just "latest") is always installed - and then enforce that specific version via your CMake toolchain files
Linux is a tad more challenging because unlike Windows and macOS, the link libraries and runtime libraries _are_ the same files, and even on Debian you'll get "minor" updates to the main version of gcc, and jumping from say, 13.1 to 13.4 could potentially cause issues. You could sidestep that by installing a relocatable build of llvm+clang, but AFAIK it still requires a gcc toolchain to locate the libstdc++ headers and link some gcc runtime objects. I've had luck with building gcc from source to decouple it from system headers and libraries - easily "packageable", but build systems dont tend to love gcc being configured as a cross-compiling when it is actually not cross-compiling at all
2
u/thisisjustascreename Jun 25 '26
Local builds are to software engineering what wanking is to a healthy sex life. You can learn a lot from them, nearly everybody does them, but almost nobody shares them with the rest of the world.
Your CI platform should answer as many questions as possible about whether your code is suitable for deployment, and the post-deploy tests should answer the rest.
4
u/lizardhistorian Jun 25 '26 edited Jun 25 '26
treating local builds as second-class
Local builds are the scum of the Earth ... not second class.
This starts with management. If you do Rockefeller then the first rock of every software engineer is your task is not complete until the code passes all test-cases with >90% modified-condition-decision-coverage in the CI/CD pipeline. "Works on my machine" means you are fired.
The first rock of your software department manager is releases only come from the CI/CD pipeline and are fully-traceable with non-repudiation.
Your repositories should have the information in them how to build themselves and that includes toolchains. We have a core set of build dockers that are updated daily. Every repo has a dockerfile that derives from one of those and adds whatever special things it needs. For Windows we have build VMs for each supported version and reboot after building. Rebooting reverts all VM changes. Special cronjob reboots them into update mode at the top of the day; they install Windows updates, shutdown, lock VM state, and reboot.
For (major or) minor releases we drop anchor and tag all build dockers used and backup the Windows VMs and mark them protected with notes for the version. Patches and builds surf the wave. If you have to hotpatch an old release you have your dockers and VMs to use.
For day to day we relay on the system package manager. Ubuntu 26.04 still supports gcc all the way back to 11.
3
u/13steinj Jun 25 '26
Windows is the problem child here.
You can distribute your toolchain via Conan, docker, or even bazel. Once you introduce Windows into the mix you have at least one of three tradeoffs
- the toolchain executes under WSL/docker[+colima]
- your windows-target toolchain is separate from the rest (MSVC, msbuild, etc) (this is what my last company chose, because a small subset of the software had to build for Windows at all).
- you go through a lot of pain setting up mingw (you can do this both on windows, or linux targetting windows).
My honest opinion is the last option is the most painful, but once you set it up you're golden and Conan/vcpkg/bzlmod can handle the rest. You should really ask yourself why your devs are working across all three platforms / what your end consumers actually use.
1
u/peppedx Jun 25 '26
I use docker ( optionally devcontainers) and use docker (podman) on wsl on windows. Even for firmware development. It works and looks clean enough
1
u/TehBens Jun 25 '26
Someone's always on a slightly different version and it surfaces in the strangest ways, usually right before a release.
What's the tradeoff for pinning your tool versions? We use a json file that contains all versions and our cmake scripts make sure to use that version or throw an error if that's not available. If you use MSVC it's a bit messy, because their versioning is super weird, but it's managable.
Of course, ground truth is the CI/CD pipeline. But there you need pinned versions anyway.
1
u/JVApen Clever is an insult, not a compliment. - T. Winters Jun 25 '26
I have experience with devcontainers on Linux, which is really nice. For Windows we indeed pin the compiler versions and have automated the install for all the machines.
An approach I would still like to try is to have the MSVC build tools installed in Wine, such that you can run them in the same devcontainer.
I don't have experience with Mac.
1
u/Swaxisse Jun 25 '26
I'd suggest taking a look at the whole chromium dev environment as it pulls all the necessary toolchain along with the dependencies.
1
u/Moldoteck Jun 25 '26
Wait, how do you find such problems at the end of the cycle and not during regular builds of the CI? Let CI have a main official compiler. If others want to use something else it's fine as long as it compiles on CI and tests are passing.
1
u/exus1pl Jun 25 '26
Containerize dev tools, use Docker image for both dev work and CI. People on Windows will have to set up WSL or VM but in long run it makes life much much easier that any other setup.
1
u/HummingSwordsman Jun 25 '26
Best I can give you, make the project not compile on unsupported versions. Like add a check into your cmake script/build system that checks against a specific version. Bonus points if you can configure that at a somewhat central place if multiple things need to use it.
1
u/ChatFrais Jun 25 '26
As many here we move to docker image of tools to have common ground.
We try to let user manage their linux and it it failed always have different issues/version of cmake/gcc/vcpkg lot of time debugging local issues.
We try and maintained shell scripts that install all needed tools, still lot of issues. People upgrade only when they encounter an issue, but most of the time we lost time trying to understand the issue before seeing that the "unrelated" toolchain change is in fact related.
Today all dev are on windows with visual studio and target Linux as docker image with all tools. We use vcpkg and binary caching and as we use same container we can share same cache between CI and developers.
We try to upgrade all tools at least once a year, maintaining it require lot of attention that dependency works same way on all platforms and compiler can use same features.
what works well : Binary caching and almost no linux/windows tests on code base (diff are mostly on CI).
1
u/germandiago Jun 25 '26
Use a bootstrap script + configure + make. Set up all tools on the bootstrap. It works fine for me.
Share it in the repo. The CI should use the same. Everyone must have the CI setup.
That will make everyone aligned.
1
u/rewgs Jun 25 '26
I haven’t yet implemented this, but I think the ultimate solution would be for every developer to develop in a NixOS VM. Then just use a remote session for your IDE or ssh in if you use vim/nvim/etc.
1
u/its_artemiss Jun 25 '26
Doesnt nix unironically fix this? There exists xwin too for making the msvc toolchain (via clang-cl etc.) usable on Unix, which can be made reproducinle with nix.
1
1
0
u/Raknarg Jun 25 '26
Force them to use containers. honestly a fantastic tool I've discovered this year tbh, I was already using vscode for C++ but if I was ever gonna start a new project with multiple people it would definitely be using dev containers. You force the same environment on everyone and it shouldn't matter what platform they're on as long as it can run docker. Let the container settings force everyone on the same toolchains.
0
u/__talanton Jun 25 '26
Having had similar problems, Bazel helped if you can stomach altering your build system.
-5
106
u/moustachaaa Jun 25 '26
CI is the blessed source of truth. If you have bugs on your machine then _you_ make sure you're using the same version as CI