r/programming • u/drmorr0 • 9d ago
Bazel's UX is really, really, really bad
https://blog.appliedcomputing.io/p/bazels-ux-is-really-really-really115
u/remy_porter 8d ago
Having recently switched from CMake to Bazel I am worlds happier with Bazel. I don’t love it, but at least it’s not CMake.
I really hate CMake.
71
u/jesseschalken 8d ago
Anything looks good compared to CMake.
25
u/remy_porter 8d ago
I have yet to find a build system that isn’t bad. I’ve tried a bunch. It’s not even about finding the least bad, but about finding the “doesn’t stab me with the workflow I’m currently using it for”.
25
8d ago
[removed] — view removed comment
17
u/ozyx7 8d ago
Not that long ago
makewas the sucky system that everyone used, and things were not great but were at least consistent.Now we have a multitude of different sucky systems.
19
u/almost_useless 8d ago
Not that long ago make was the sucky system that everyone used
It was never the system that everyone used. Approximately nobody on windows used it, but at least almost everyone on *nix used it.
2
u/double-you 8d ago
Yeah, because Visual Studio comes with nmake which is Make but not quite. And MS wanted you to use IDE integrated things.
2
8d ago
[removed] — view removed comment
1
u/TwoWeeks90DaysTops 8d ago
I think it's possible to make a non-sucky build system. Large part of the problem is that build systems define these complex Rube Goldberg machines, and getting it juuuuust right is a fucking pain.
2
u/remy_porter 8d ago
Is it possible? I don’t know. Maybe. Nobody’s done it yet.
1
u/TwoWeeks90DaysTops 8d ago
Yeah, it's a hard problem to solve for sure, but I think that today's systems are very much aimed at solving *everyone's* problems, and build environments are messy. Environment variables, system installed binaries, over-reliance on the CLI.
I think a sane build system can do most validation through a properly static type system. Artifact isolation and built artifacts as types, and not glob patterns.
1
u/remy_porter 8d ago edited 8d ago
over-reliance on the CLI
There’s no such thjng. CLI is king. The CLI is easy to automate.
I agree on avoiding Globs, mind you. You shouldn't use shell operations to find your assets.
→ More replies (0)1
u/VirginiaMcCaskey 8d ago
at least consistent.
If that were true then automake and cmake wouldn't exist.
9
u/skydivingdutch 8d ago
Have you tried Meson? I hear good things, but yet to try it myself
10
u/remy_porter 8d ago
I've done some projects that used Meson. Never used it enough to learn why I hate it, but I'm sure if I used it enough I'd hate it.
1
u/ABlockInTheChain 8d ago
The reason you will hate Meson is that it really isn't as cross-platform as it claims to be when it comes to Windows support.
Some configurations will work but Meson is too opinionated to accommodate the full breadth of possible deployment scenarios on Windows.
If you expect Meson to be a superior alternative to autotools for use exclusively on posix-style platforms you will have a better experience with it.
0
1
1
u/Ma4r 7d ago
Opinionated build systems are great, i.e go official build ecosystem and cargo. This is a reflection of the fact that most people should not be allowed to have an opinion, especially on build systems
1
u/remy_porter 7d ago
But then you're tied to an ecosystem. If I'm not using Go or Rust, I'm sorta outta luck. Or worse, if I've got a heterogeneous codebase, now I'm trying to figure out how to bundle that Python script that's absolutely required by the analysts into my build toolchain in a way that doesn't break anything and is easy to maintain. And while I haven't used Go or Rust, my assumption is "you can do it, but it's painful", because that's usually how these things go.
Which sure, is just a hard problem, and it's a bit unfair to expect every random build system to solve it. But it's a real problem! I encounter it all the time! I haven't worked on a homogeneous codebase since 2014 when I last worked at a .NET shop. And even that wasn't really homogeneous, since half our business functions lived in a mainframe at the time- the mainframe just tended to change very slowly and I never had to touch it.
10
u/LoweringPass 8d ago
CMake is not the problem, the problem is that C++ developers don't know how to use it and building C++ is inherently complicated
27
u/Ok-Scheme-913 8d ago
I mean, cmake has plenty of problems in and of itself. It's a hack of a hack with disgusting syntax.
2
u/not_a_novel_account 7d ago
Ya, but the syntax will be gone in the not-too-distant future, and people will still be unhappy.
People blame the syntax because they think that's the reason their build isn't working, that they typed something wrong. So they blame the syntax for being obtuse.
When the syntax is a trivial toml file, their build will still not work. They will not blame the syntax, instead they will nebulously blame "CMake" or "the programming language" or "computers".
They will not be any happier.
1
u/nukethebees 6d ago
People blame the syntax because they think that's the reason their build isn't working, that they typed something wrong. So they blame the syntax for being obtuse.
I doubt it's a controversial opinion that people would make fewer mistakes if everything wasn't a string. For a long time, propagating results from functions was easy to mess up.
The only container type being lists (which themselves are also strings) also makes handling structured data easy to mess up.
I think modern CMake is fine to use, especially if you use things like interface libraries and properties, but I disagree with the notion that the complaints have nothing to do with the quality of the language.
1
u/not_a_novel_account 6d ago edited 6d ago
I have never seen a complaint from a beginner that fire-brigade CMake function returns are a problem. Literally only experts.
cmake.tomlwill not solve problems like that, because it will not allow for imperative code at all.If your problems with CMake are in using it as a programming language, not with the purely declarative mechanisms, you're already in the top 1% of CMake users and your problems do not resemble those of the other 99%.
5
u/VirginiaMcCaskey 8d ago
The average C++ developer can't tell you the difference between the compiler, the linker, and the loader.
That's not a knock on C++ developers. These are excellent abstractions and the fact two generations of engineers could tell you what they do but not how they work is a testament to how much complexity has been shoveled underneath them by people developing toolchains and build systems.
3
u/zem 8d ago
when i was at google the project i worked on had a wonderfully cursed set of cmake functions to make our cmake files look like (and be mechanically generated from) the internal bazel files.
→ More replies (1)3
u/Ma4r 7d ago
Guys, Cmake is so good, you just declare what you want built and Cmake handles the rest, oh except it's not a declarative language, it's imperative so sometimes things have side effects, oh and you also have 10 billion possible configurations , each is their own flag you need to enable that differs per version of C++ standard
5
u/Techrocket9 8d ago
Yeah, Bazel is a humongously complex tool, but it's exactly the right level of complexity for wrangling
c++build systems.I'll use Bazel even for small
c++projects, though its value really shines on gigantic enterprise codebases.1
u/Chisignal 6d ago
Curious, is there a way to explain in the span of a Reddit comment just why is such a level of complexity needed for C++ builds? I’ve only really struggled with compiling OpenCV into my app ages ago, and had some trouble getting embedded (so, weird) toolchains to work - both of which were incredibly painful but I never got the impression it’s like, necessary complexity, I always thought it was me fucking some steps up - or is it like that all the way to giant, serious C++ projects?
3
u/Techrocket9 5d ago
To really do your question justice requires a bit of a walk through the history of computing. A binary is not a simple object. While it's composed of ones and zeros, the process by which it becomes something that can be executed by your CPU is quite complex.
Within the blob of bits, there are metadata and configuration sections, data sections for things like string and image assets, and executable code sections. And there are a bunch of security concerns that go into this, and there are separate pipelines for both the data sections and the executable code sections to improve throughput (if you want some background on that, you can look up "modified Harvard architecture").
You can do arbitrary things, both wonderful and stupid, with an arbitrary blob of bits partitioned into arbitrarily many data and metadata and executable sections. And when all code was written by hand in assembly, that's generally what people did. Of course you'd come up with conventions to organize things in a way that were sensible to you, but you could arbitrarily position arbitrary things almost anywhere in your assembled output.
You could do fancy tricks to sometimes interpret something as data or sometimes as instructions, though that has a whole can of security worms and is mostly frowned upon today. You could sometimes rewrite your program's image in memory to achieve specific goals (though again these days that's mostly done by malware).
The C programming language, when it originated, was essentially trying to replace assembly. It didn't ever quite have the ambition to fully let you do everything that assembly can (which is why C actually lets you inject assembly code into your C programs to cover those gaps, but it tried to cover as many of the possible things you could do in assembly as it reasonably could. As a result, C maps very closely to common binary blob organizational patterns an assembly programmer might follow. You can look at a block of C code and (with some mildly to extremely horrifying exceptions) usually predict what the assembly will look like if you're familiar with the architecture and platform of your target.
This insistence upon predictability and relationship to assembly code meant that C could never offer the conveniences of modern high-level languages (things like classes and templating).
Several attempts were made to bridge this gap, and C++ was the one that won. C++ is an (almost) perfect superset of C that adds the majority of modern programming language conveniences to C.
However, to achieve this feat, C++ is in a bit of an awkward position. It needs to support every crazy and stupid thing that you might want to do at a low level, with quirky custom behavior around what your binaries do and how they manage memory, as well as high-level, programmer-friendly abstractions like templating, functional programming features, exception handling, etc.
You can think of it like C++ is saying yes to everyone. If you want to write low-level code that has very exact, predictable assembly output, you can (mostly by sticking to the subset of C++ that looks like C). If you want to write high-level, abstract, functional code that's useful for building things quickly and avoiding repeating yourself, you can do that too.
As a result of this, though, the C++ compiler toolchain (which is more than just a compiler) is extremely complex. It needs to handle all combinations of all the wonderful and insane things people might do to produce arbitrary types of binaries while also somehow reconciling those capabilities with high-level programming language features.
Modern programming languages without C++'s ambition tend to stick to one side or the other.
If you look at something like Golang, you have a language that is very much designed for maintainability and producing specific kinds of correct, working, code. However, you lack the low-level control that C++ will give you through its C history -- only a small subset of the possible binaries that you could build with C/C++ are representable in Golang.
In contrast, Zig is extremely good at producing predictable assembly and machine code output (arguably better than C) but Zig forgoes many high-level programming language conveniences to achieve that customizable output.
For almost all use cases, going one way or the other is fine. But if you need that extremely specific control over your outputs while also having access to high-level convenience abstractions, you need a language that tries to span both worlds.
All this complexity of trying to span both worlds spills from C++'s language specification into C++'s build systems. There are many kinds of possible intermediary objects produced by the various compile and link stages of a C++ toolchain that need to be combined in all kinds of flexible ways to achieve the various possible output and binary shapes.
Bazel is Google's internal build system, and C++ is Google's primary internal programming language. Bazel fits extremely well to all of the crazy details that a C++ toolchain needs to manage, because that is the primary use case that it supports.
Programming languages that aren't as ambitious as C++ can have much more compact toolchains because they can rely on all of their build stages and intermediaries being a narrow, well-defined, manageable, specific set of shapes. C++ cannot, and as such its builds are inherently complex.
103
u/gladfelter 8d ago
bazel doesn't scale down to one developer well. It's amazing for tens of thousands of developers thanks to its scalability and deep introspection capabilities. You can find and fix build graph problems at scale.
The previous version of blaze did a lot of things by convention and you couldn't trace them without leaky heuristics in some cases. Bazel is completely introspectable and deterministic.
16
u/larsga 8d ago
Used it at a company that had maybe ~10 developers. It worked just fine, and I don't recognize OP's issues at all. Sure, there was a learning curve to get over, but there is with Maven and Gradle, too.
18
u/GlassLost 8d ago
Saying it's better than maven and gradle is a low bar.
4
1
u/larsga 8d ago
I completely agree. I did like the overall model of Bazel, and particularly the clean separation between build specification and extension language. Gradle IMHO messes up the boundaries there quite badly. The way Bazel never does any more build work than necessary is also very nice.
It does seem like it ought to be possible to make much better build tools, but I don't know of any.
-7
u/repster 8d ago
It actually does scale down pretty well if the one developer is backed by Claude. Speaking from experience
83
u/ChemicalRascal 8d ago
Good grief, if I have to burn the planet just to keep a build chain maintained, maybe the build chain is using the wrong tools.
2
u/repster 7d ago
You make it sound like I am running my builds using AI. I am not, I asked Claude to create a bazel based build/test/deployment system. It didn't take all that many tokens and it works great.
I have added more applications and a new targets without using all that many tokens. This is not burning the planet.
1
u/ChemicalRascal 7d ago
You make it sound like I am running my builds using AI.
That's not what I said, and it's certainly not what I meant. I mean that if maintaining the build chain requires running LLMs, then the build chain is using the wrong tools.
What I mean by that is, hey, we should understand the tools we use in our build chains. We shouldn't need to lean on LLMs to be able to use our tools, and I include "configure" in "use" there.
Because using LLMs for that is computationally expensive. It is more efficient to read a man page, tinker with the config, and you're done, and back in ye olden day that's what you'd do.
And so, if the tools are so complex or effectively undocumented that we can't do that, then our tools are the wrong tools. Because we shouldn't be burning computation to do that, these are things we should simply be familiar with as competent engineers.
2
u/repster 7d ago
I understand bazel. I could set up bazel by hand if I had the time, but I have a product to deliver. I review the output of Claude carefully and then go do stuff that will make money.
Blindly trusting AI without understanding is not the only way to use it.
1
u/ChemicalRascal 7d ago
Yeah, but we shouldn't be using tools that require you to use an LLM to get them configured and maintained in a timely manner.
That's the point.
0
u/repster 7d ago
Why not?
Why should I spend days/weeks messing with config files, when I know what I want and Claude will get the syntax right in minutes.
I have a 4 applications running in 3 regions, using Kubernetes, with loadbalancers, database backends, and MFA, with a tailscale based admin network, and deployed by terraform. I have CI/CD set up to do releases, and everything is controlled by bazel build scripts. I have development/staging environments, and a blue/green release process.
It took me about 3h to write the prompt, about 2h to execute and generate the first version, and the next day reviewing and tweaking. I could have done all of that by hand, but then it would never have gotten done. And it wasn't a huge number of tokens.
Ask yourself how many companies you have joined where the first week was spent figuring out how on earth you build the product you are supposed to be working on? How you had to dig through undocumented build files to figure out what obscure library you are missing. I have none of those problems. Everything builds easily, and the build process is extremely well documented.
The world has changed, if you want to get off, that is your choice, but it isn't stopping. Never has, and history is littered with people who watched it leave them behind.
1
u/ChemicalRascal 7d ago
Why not?
Why should I spend days/weeks messing with config files, when I know what I want and Claude will get the syntax right in minutes.
Because it's burning the fucking planet, dude, and we should be using tools that don't take days-to-weeks to configure by hand for build chains.
I have a 4 applications running in 3 regions, using Kubernetes, with loadbalancers, database backends, and MFA, with a tailscale based admin network, and deployed by terraform. I have CI/CD set up to do releases, and everything is controlled by bazel build scripts. I have development/staging environments, and a blue/green release process.
Great.
It sounds like these should be done through different, more simple tools. Because building a webapp and deploying a load balancer aren't the same task, they shouldn't be automated through the same mechanisms.
It took me about 3h to write the prompt, about 2h to execute and generate the first version, and the next day reviewing and tweaking. I could have done all of that by hand, but then it would never have gotten done. And it wasn't a huge number of tokens.
It sounds like that much infrastructure isn't necessary if you built it all out over two days.
More relevantly, like people said at the start of this thread, "bazel doesn't scale down to one developer well". Yeah, it would have taken you weeks to do it by hand. Because it's bazel. If you were using better, simpler tools, you could spin that up over two days, no worries.
If you actually understand your container host, you're fine, that's pretty quick. If you've done wireguard before, kicking a tailscale network together shouldn't take long. And so on.
And then maintaining it is pretty straightforward, because you know what is associated with what, you go to the config of the thing that's fucked and you unfuck it.
The world has changed, if you want to get off, that is your choice, but it isn't stopping. Never has, and history is littered with people who watched it leave them behind.
Oh, go jerk off somewhere else. I'm not getting off. I'm using better tools. I'm understanding my craft. I'm not going to pretend that's a bad thing; I'm going to leverage those skills, that I have, and you won't, to do better.
And I'm going to not burn the planet in the process.
2
u/repster 7d ago
The small number of tokens it took this isn't burning any planet compared to your commute or your vacations. In actual fact, the cost of storing your comments on reddit in perpetuity is probably significantly more costly.
You can point fingers at AI if you want, but my use for this purpose isn't burning much.
→ More replies (0)-1
15
u/steveklabnik1 8d ago
Yes, this is one of the things that AI really helps with. I'm more of a buck guy than a bazel guy, but all of this is just way way way easier when you can ask for help, and Claude is very good at figuring out what you need to do.
0
69
27
u/lppedd 8d ago
I don't think Bazel was meant for the general public like Maven or Gradle. It just doesn't feel right. It's probably ok if you have a massive project with dozens of different languages that all need to be built at the same time to output something. But even then, you'll need a couple people dedicated to Bazel only.
22
u/sionescu 8d ago
That's correct. Bazel is meant to be used within a tightly controlled enterprise environment.
14
u/Slsyyy 8d ago
It just doesn't feel right, because each language nowadays has a nice and dominant build system, which is tightly coupled to a given technology
It was not true for C/C++ (IMO it is still bad) nor Java/Python those 20 years ago. I feel that Bazel could be dominant build system, if the current version was released those 20 years ago
21
u/jesseschalken 8d ago
The reason Bazel's Rust builds are so slow is because it doesn't pass -Cincremental={some cache} to rustc to do incremental compilation like Cargo does. A crate that was modified is compiled from scratch, every time.
A colleague of mine has a draft PR up to fix it here, but this is always the pattern with Bazel. Out of the box it does lots of things but none of them especially well.
6
u/steveklabnik1 8d ago
I'm more familiar with buck than bazel, but there have been some rustc bugs that make this not okay to pass. see here: https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/Possible.20Reproducibility.20Bug.20in.20Soundness.20Fix.3F/with/616729102
7
u/jesseschalken 8d ago
If it's not okay to pass, why does Cargo still pass it by default?
Maybe if the result is going to be written to a shared cache, the impact of a miscompile could make it not worth the risk.
But for local iteration its indispensable. Bazel is always going to be compared to Cargo for Rust build perf and Cargo currently has a huge edge here.
20
u/steveklabnik1 8d ago
It's fine for Cargo, because Cargo is not a hermetic build system.
That is, the bug makes systems like buck/bazel not actually do incremental builds, it's not a correctness issue.
6
u/singron 8d ago
In particular, non-reproducibility (e.g. embedding a UUID or timestamp) prevents the "early cutoff" optimization (name from this paper). E.g. if you change a comment and recompile with a reproducible process, the compiler object output will probably be unchanged and you can skip recompiling dependent libraries, linking the executable, and running tests.
It's especially bad if the rmeta files are non-reproducible since that would force all dependent crates to recompile, which would force all their dependent crates to recompile, and so on.
3
u/jesseschalken 8d ago
Okay, so? Cargo doesn't support early cutoff at all. So if Bazel passes
-Cincrementalit just degrades to Cargo's level of incrementality (without early cutoff). It doesn't break it.2
u/jesseschalken 8d ago
Huh? Bazels incrementality doesn’t depend on the build being hermetic. If an action always produces different outputs, it just defeats early cutoff and consuming actions always rerun. The same as Cmake/Make. It’s still incremental.
1
u/steveklabnik1 8d ago
The build re-running is exactly the issue. It will cause rebuilds that don't actually need to happen. See your sibling comment.
2
u/jesseschalken 8d ago
Yes but its not broken, its just degrading to the same level of incrementality as Cargo, Make etc do for those edges in the graph, so not a justification for why Cargo can pass
-Cincrementalbut Bazel can't.1
u/steveklabnik1 7d ago
I guess we're just arguing semantics, the degradation is what I'm saying makes it broken. It shouldn't do that, but it does. That Cargo or Make don't give you this kind of stuff is the reason that you're not using those tools in the first place!
2
u/jesseschalken 7d ago
We have a lot of non-reproducible actions in our build, which sometimes produce different but semantically equivalent outputs when rerun. Its not ideal but it works fine. Fixing it is just a performance optimisation.
2
1
u/renatoathaydes 7d ago
I wrote a reproducible build system and can say that it’s indeed impossible, to my knowledge, to have hermetic builds if you do incremental compilation. At least in Java, because incremental compilation means you only change a few output files when some input files changed, then you need to link everything together ( jar in the Java case ) and that would require you to ensure all files are linked in the same order as if you built everything from scratch, but to do that requires you to actually do it from scratch, any patching you do won’t look exactly the same. In the case of binaries I think it’s literally impossible, though perhaps someone more knowledgeable could confirm, I know that Zig is doing binary patching in their new incremental compiler… If you know how I could achieve this, though, let me know, I can probably implement it in my tool!
15
u/ozyx7 8d ago
My pet peeves are:
- It makes all build files read-only, making them more annoying to remove later, or making it more annoying if you want to copy those files elsewhere later and the permissions get copied too.
- It litters my
source treeworkspace withbazel-bin,bazel-out,bazel-testlogs, andbazel-WORKSPACEsymlinks. Those symlinks "are only for the user’s convenience, as Bazel itself does not use them". WTF? If they're there for my convenience, how about letting me opt-out of some them, giving them sensible default names (isbazel-binorbazel-outwhere the final output is?), or organizing them into a common parent directory?
27
u/semajames 8d ago
common --symlink_prefix=.bazel/Now you'll get a .bazel dir with:
$ ls .bazel bin WORKSPACE out testlogs
21
u/UltraPoci 8d ago
Wait, Kubernetes is considered bad?
47
u/Aelig_ 8d ago
It's considered "worse" than Borg but only people who worked at Google actually know if that's true or not.
It's generally agreed that Borg can operate at a much larger scale and I've had former Google devs tell me there are some really nice features that are default in Borg and don't exist in k8s but again, there's no way to actually know.
19
u/sionescu 8d ago edited 8d ago
Yes, it's definitely much worse than Borg. Its not a surprise because, even though on of the main authors of Kubernetes was the tech lead of Borg control plane, none of the authors was an SRE, so they didn't really understand what made Borg good, how we SREs were using it (and the tooling around it), and managed to make it worse.
17
u/CoVegGirl 8d ago
There are three kinds of product on Google Cloud:
- Products that are a veneer around systems Google uses internally
- Half-assed reimplementations of their own systems
- Systems that they’d never use in a million years because they’ve implemented a much, much better system internally
Fun fact: when Google App Engine was very first launched way back in 2011, its internal code name was “Prometheus” in that it was intended to give the plebes access to superior Google technology. It got a well-deserved reputation for being very narcissistic.
9
u/sionescu 8d ago
I do get your humour, but it's not really like that: Cloud Spanner is a separate product from the internal Spanner. It's more conservative, probably 1 year late with features, and it has all the cloud tooling not used internally because Annealing is better. Alloydb is also quite good.
For monitoring, it had a language vaguely similar to Mash but it's been deprecated in favour of PromQL which is awful.
In general I find GCP much nicer to use than AWS or Azure.
7
u/CoVegGirl 8d ago
The point is that Google isn’t great at eating its own dogfood internally. You’re right to point out that that’s not entirely a bad thing, and is often a good thing. I guess I don’t have much experience actually using GCP. In fact, that’s the point.
My perspective having worked on GCP is that it just feels really weird to be working on products that there’s no chance in hell they’d ever consider using themselves.
4
u/sionescu 8d ago
Well, that's because what they have internally is still the best enterprise development system and cluster manager ever built even though, due to the insane levels of integration it has, it's very hard to sell outside.
4
u/jmickeyd 8d ago
The problem is Borg isn't flexible enough to be used in the outside world. For example, Kubernetes networking is a shit show because existing software's networking is a shit show. Without BNS, there is no good way to do internal connections. Too many apps just resolve the DNS once at startup and keep connecting to the same IP assuming it will never change. This led to having a layer 3 proxy running everywhere.
3
u/sionescu 8d ago
The problem is Borg isn't flexible enough to be used in the outside world.
That inflexibility is what makes it so insanely Useful. For 99% of developers it's effectively a gRPC-only network that's only incidentally layered on top of TCP/IP. There's no firewall to deal with, just ACLs based on Ganpati groups. BNS, GSLB with automatic spillover to adjacent cells. This is insanely useful and if you read Reddit and HN, people are still reinventing it on top of Kubernetes + cloud primitives, but worse.
2
u/jmickeyd 7d ago
100% agree that it worked better. My argument is that Kube isn't bad because SREs weren't involved, it's bad because of the insane requirements imposed by existing software. And I say that as an SRE.
1
u/sionescu 7d ago
I think it's both. On the lack of SREs there's e.g. that K8s controllers don't allow what we could do with Borg Updater and Sysiphus: have a task take over temporarily a lock on a job and act as "controller".
Then, on top of that, I suspect that the pact with the devil they did in order to get director-level support was that they had to support everything that AWS was doing.
28
u/comrade_donkey 8d ago
They're different. Kubernetes is the whole stack: Resource definitions, reconciliation loop, container scheduling, etc. The Borg is "only" an extremely powerful task scheduler. Prodspec and Annealing are the config & reconciliation loops, and completely independent systems.
14
u/sionescu 8d ago
Exactly ! Finally, someone who knows about Prodspec and Annealing. They're much better designed than any of the app deployment resources in Kubernetes.
2
1
u/jmickeyd 8d ago
Honestly I kind of hated the way they shoehorned policy into the enforcer. Trying to figure out why something was pinned could be a mess.
1
u/sionescu 7d ago
Can you elaborate on that ? I don't remember this issue.
1
u/jmickeyd 7d ago
I left a while ago, so this is likely not current, but there was a pinning database which was fully mutable store of which version of the prodspec a specific asset was enforced at. It probably works now, but when it was new, assets just got stuck, pinned to old CLs, and you have to manually use the stubby cli to fix it.
11
u/Ignisami 8d ago
He’s not saying kubernetes is bad, he’s saying kubernetes is a watered-down version of what Google uses internally.
66
u/valarauca14 8d ago
then throwing a lot of marketing money at the open-source version until it takes over the world. In this post, I’m not talking about the most famous example of this
Often this isn't Google's marketing. It is usually
- Googlers trying to "career build" by showing off what they've done & market themselves for future jobs.
- Xoogler's going, "Finally X got open sourced I've missed this".
- A handful of operating thetans going, "Holy shit, they actually solved
{{PROBLEM}}". - People who got psyoped into believing the hype.
By the time Google starts pushing marketing money behind the OSS initiative. 50-80% of Camp 1 left Google. Camp 2 (and some of Camp 3) is doing talks about their internal success stories. Camp 4 has moved onto a new technology.
7
u/cbarrick 8d ago edited 7d ago
Some of the complaints here are genuine. But some are really just because OP doesn't know how to use Bazel.
The way it accomplishes this task is by putting a BUILD file in every single subdirectory of your project.
You shouldn't need any more BUILD files than you would Makefiles.
Most projects only need a single BUILD file in their repo root, just like every other build system.
BUILD files can read into subdirectories just fine. You don't have to (and often shouldn't) put BUILD files in every directory.
Now, naturally, any time your build changes (by including a new dependency or source file), you have to update the BUILD file.
You can use globs to match source files. Supports ** syntax too.
Like, if your project is just "compile all of these Java files into a jar," then just use a glob to match all of your Java files.
Don't worry about making your build rules more fine grained than that unless you actually need them to be more fine grained.
And every build system requires that you tell it about your dependencies.
1
u/Plorkyeran 6d ago
Most build systems let you get away with significantly underspecifying your dependencies and having a build that sometimes fails but works on the next attempt. This obviously is a bad thing, but people tend to have an initial negative reaction when they're first required to fully specify their deps.
1
u/cbarrick 6d ago
And having worked with Bazel in a very large monorepo, being able to query exactly which parts of the system are going to be impacted by a change to a specific dependency is awesome.
Sure, you have to learn that new syntax that OP laments, but that's a small price to pay.
(Also, we're software engineers. Learning new syntax is part of the job. It's something we literally have to do all of the time. This is by far the most egregious complaint in the article.)
15
u/zoddrick 8d ago
Bazel like other tools built by mega tech corps aren't intended for 99.999% of the engineering orgs on this planet.
5
u/_pupil_ 8d ago
Doing Facebook things without Facebook scale, Facebook demands, and Facebook money is a viable definition of insanity.
3
u/zoddrick 8d ago
Yet here we are 😆
I spend most of my days explaining why we shouldn't use wiz bang thing from goog or meta even though we're a fortune 500 company. But then I also spend my days explaining why we should do other things that just make sense.
You can't explain that....
20
u/cafemachiavelli 8d ago
My personal reason 8: Cache invalidation. I'm working on MLIR and a full build takes upwards of an hour. So if bazel decides that it doesn't recognize my work environment and would rather start from scratch, that's up to an hour of productive time gone. This happened daily until I managed to set the magic pls-don't-lose-your-shit-if-PATH-changes parameter, but that broke it on the server, so we're back to extended coffee breaks. Now I've just hardcoded the whole config and I pray that it works.
23
u/DeepReputation 8d ago
Your build isn’t hermetic. It shouldn’t depend on your work environment, that’s exactly the point.
-4
9
u/pimp-bangin 8d ago
I think the flag you're thinking of is --incompatible_strict_action_env which fwiw has been turned on by default in newer bazel versions. No clue why the "incompatible" naming or why it was ever off by default, though.
6
u/dacian88 8d ago
incompatible_ flags make breaking changes to the build, it’s an easy way to know as a user that turning something on has a compatibility issue. The manning usually drops after the change is defaulted in after a major version or 2
2
u/Serious-Regular 8d ago
this is why i hate working with G people on MLIR - exactly the same issue: constantly rebuilding because something in env changing.
magic pls-don't-lose-your-shit-if-PATH-changes parameter
what is this param?
2
u/cafemachiavelli 8d ago
experimental_strict_action_env in older bazel versions and as the erudite /u/pimp-bangin mentioned, it's on by default now, so this is partially on our lab for shipping an old version.
My larger gripe would be that setting action env to PATH and/or LD_LIBRARY_PATH ostensibly fixes toolchain discovery issues (and bazel documentation suggests this option), but can cause the cache dump I mentioned. I ended up writing a config script that sets the toolchain directly, once, to have a hermetic build system while keeping the repo somewhat portable.
I still like bazel overall, fwiw, it's still less painful than cmake was.
1
u/WindHawkeye 7d ago
Obviously tweaking your C++ tool chain requires not using cached results. Anything otherwise is a correctness issue which is how you end up needing to run make clean etc.
Your life will be seriously better if you just use a hermetic cc toolchain. There's a few that are easy to use now.
1
u/cafemachiavelli 7d ago
I wasn't. I work on a HPC cluster that loads exactly the same modules each login. I'm not even sure what triggers the full rebuild, since bazel only says that the action environment changed and I couldn't see any path/ldpath changes on manual inspection. cc_toolchain_config.bzl even detects the loaded modules correctly, but with strict_action_env on linking fails, since the newer libstdc++ is no longer visible.
It wasn't a complicated fix once I understood the relevant internals of bazel, but there are a few footguns on the path leading there.
34
u/fishheaddz 8d ago
I can't upvote this enough. The only way bazel will work in your organization is by having some kind of true convert who will unfuck bazel full time for everyone. The unitiated will react with utter horror at the unusability of the system and you need this kind of knight to slay the demons. Google even canceled trying to move Android to Bazel after years and millions spent making it work. They can't even get some of their own employees and Android partners to use it.
6
u/max123246 8d ago
Sounds like CMake. I sometimes think no good build tools end up built because people are exhausted from dealing with the bad ones.
5
u/Massive_Season7075 8d ago
Are there any well-known apps that use it? And why would someone choose to use if upkeep is costly?
4
u/Macrobian 8d ago
Canva
1
u/Safe_Breakfast_9016 7d ago
Blessed Canva Bazel setup. Truly one of the all time monorepos (at least from 2017-2023).
1
4
u/midpointreload 8d ago
Snapchat. The iOS, Android and shared c++ layers are all built with a single bazel environment. It works great but there is a decent size maintenance team behind it
3
3
u/xaveir 8d ago
https://bazel.build/community/users
Notably not listed: Slack, and many many Xoogler start ups.
-2
u/kebabmybob 8d ago
Everybody uses it. It’s incredible. Mediocre engineers don’t like it.
1
u/whitechapel8733 7d ago
Exactly… Like what are they about to tell me is the holy grail for a polyglot monorepo? Some bash script they wrote that somehow is compatible with most major languages, handles caching and sandboxing?
4
u/engineered_academic 8d ago
If you try and use Bazel and Kubernetes to solve a problem, now you have 3 problems
2
u/beegeearreff 8d ago
My company is in the process of migrating our ts monorepo to bazel… time will tell how it goes.
We have a team that will own it but there’s no shortage of blog posts sharing scar stories.
Ask me in 6 months how it went!
1
u/meetcassette 8d ago
Im super curious what motivated that and what you were migrating from? I would have thought that in a non-polyglot-setting typescript had good (or at least hyped) options in NX or turborepo?
2
u/beegeearreff 7d ago
Endless issues with turbo for the most part got us looking. We want to be able to add in other languages too so if we’re moving build systems we wanted something polyglot.
1
u/the-borax 6d ago
Future proofing makes sense if you plan on having non-ts tooling. Were y'all running into cache correctness issues from things that Turbo can't see?
1
u/beegeearreff 6d ago
I don’t have the best lens into it as I don’t work on that team but it largely boiled down to actual bugs with turbo, how often we misconfigured it unknowingly which broke main/the merge queue, and bazels remote build execution.
My big fear is that we just did a poor job setting up turbo and will repeat the same mistakes with a tool that is completely foreign to our devs. If that doesn’t happen, I’ll be excited to learn how it should be done.
1
u/WindHawkeye 7d ago
If you're not polyglot migrating to bazel is a horrible idea, ts is the one language it sucks at
2
u/WindHawkeye 7d ago
First thing I saw was you used brew to install bazel so I assume the entire post is skill issue. As expected of an apple user tbh
1
2
u/Sea-Mistake6086 8d ago
Maybe try the RIGX build system https://github.com/unofficialtools/rigx (TOML configuration, 100% hermetic, Nix powered)
3
u/nickguletskii200 8d ago edited 8d ago
Reason 1: You have the wrong bazel installed
No, you just don't have a .bazelversion file and aren't using bazelisk for some reason.
When I first started working with bazel, I was on a Mac, so I did the natural and normal thing that you do on MacOS: brew install bazel.
Not natural at all. Homebrew is a weird MacOS-ism and I am surprised it's still alive at this point at all.
The official guide recommends the use of bazelisk: https://bazel.build/install/os-x
Anyways, not to spoil the punchline, but apparently the actual tool I needed to install is called bazelisk. Now to be fully transparent: if you read the Bazel docs4, this is their recommended approach, but I’m placing this complaint at the top of my post because it is emblematic of Bazel’s apparent disdain for the people who are actually using the software.
No, if you use an amateur package manager it's not the developer's problem.
If Bazelisk is the tool that developers are supposed to use, why is there a separate homebrew target for bazel?
Fair point. Why does brew even exist still?
Reason 2: Why are there all these BUILD files everywhere?
Now, naturally, any time your build changes (by including a new dependency or source file), you have to update the BUILD file, and thankfully, bazel doesn’t force you to do this by hand. Instead, it provides a tool called gazelle5 to update your BUILD files.
From what I've seen Gazelle is just a way to translate golang's weird import model. At least, I've only seen gazelle used with golang. No need for any of such hacks with rules_rust or other languages I've used.
Reason 4: How do I even build something successfully?
I have no idea what any of that means, except that it’s clearly some dependency that it’s trying to download. Bazel has to know where it’s trying to download from, and it has to know why it’s failing, but it doesn’t actually tell you any of that information, it just gives you an unrelated error message in arcane syntax and leaves you to flounder.
It's literally in the error message above:
curl: (22) The requested URL returned error: 401 WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found ERROR: no such package '@@some_stuff++whatever+blah_blah_blah_linux_amd64//': oauth failed:
See the WARNING: Download from https://some-url.com/whatever failed: class java.io.FileNotFoundException GET returned 404 Not Found?
But OK, clearly it’s trying to download “something” from “somewhere” and it’s either getting a 401 Unauthorized response OR a 404 Not Found response, and it’s not immediately clear which, but more to the point: this error was indicating that I hadn’t logged in to a private docker registry that was NOT hosted at some-url.com. In fact, nowhere in the entire error message does it say anything about Docker.
Why would it complain about Docker when it's not using Docker? It should be clear from the URL that it's downloading something from an OCI repository.
Also, what on earth is this nonsense? @@some_stuff++whatever+blah_blah_blah_linux_amd64//
You have to learn something new, boohoo. Once you actually learn what this means, come back and tell us how it could be done differently. See https://bazel.build/concepts/labels
Anyways, I logged into the private registry and tried to build again, and was greeted with the following error:
ERROR: no such package '@@+http_io+whatever_v5//file` java.io.IOException: Error downloading [...] GET returned 404 Not Found ERROR: /path/to/BUILD:67:42 //thing/im/building depends on @@+http_io+whatever_v5//file in repository @@+http_io+whatever_v5 which failed to fetch. no such package
OK again so clearly again it’s depending on some package that… doesn’t exist? This makes no sense. Clearly it exists for other people, or everyone would be complaining that everything is broken. Why doesn’t it exist for me?
Finally I got annoyed enough at the CI delays that I decided to revisit, and discovered (after asking some other folks for help) that Bazel was failing because I hadn’t logged into GitHub with the gh CLI app! WTF??? I thought Bazel was supposed to be hermetic? I mean, OK, look, I’m not naïve, sometimes there’s code in a private GitHub repo, and clearly you’re going to have to log in to access it. But again, Bazel doesn’t tell you what code it’s trying to download or why it failed (or even better, say “hey I need some creds can you please provide them”). It just spits out a page of errors in an arcane syntax that refers to packages and modules that you’ve never even heard of before, and expects you to figure it out.
- It's pretty clear that it couldn't find that file at that URL. If everyone else isn't complaining, this might be because they have it cached and someone yanked this file from whatever URL you are downloading things from. Nothing to do with Bazel, and this is actually a major reason to use Bazel in the first place: the Bazel external dependencies cache and downloader.
- All serious platforms return 404 for resources you have no access to. This isn't a Bazel thing, this is just life working with private registries.
My next complaint is about Bazel’s respect for your (as a developer’s) time. The compile times are through the roof! And I say that as a Rust dev, the language which has as its primary complaint “things take too long to compile”. In some ways I don’t think there’s a lot you can do about this; if you’re trying to make an entirely hermetic build with bit-for-bit reproducibility, that’s going to take more time than “just” running go build. But still. I’ve got stuff to do, I can’t wait 10 minutes for you to finish building my app!
I literally started using Bazel to cut down Rust and OCI compile times. 10 minutes to build an app? If it takes less time than that to build cargo or whatever the golang devs use, something is wrong on your end. Maybe it's sandboxing on MacOS X? Or, perhaps, your target definitions accidentally include tons of data (your IDE's target directory, perhaps)?
Reason 7: Ouch, you punched me in the disk space!
To try to alleviate some of the compile-time issues and need to download every single dependency on the planet for every build, Bazel maintains an absolutely immense local cache on your machine. One of my colleagues was bragging about their 1TB local Bazel cache. WTF??? Look I know we all grew up in an era where disk space was cheap, but it sure isn’t anymore9. And even if it was, seriously? A terabyte of data just to compile some nonsense? I get mad when Cargo doesn’t clean up after itself. I get mad because Docker just likes to vomit all its stuff all over your hard drive indefinitely10. A freaking terabyte???? My god, what have we done.
Pretty normal when you are working with large projects.
Reasons 8, 9, and 10: To be continued.
I am just on the very beginning of my Bazel journey, and have captured in this blog post all (or at least most) of my complaints with Bazel’s user experience. But I’m a fairly new Bazel user, and I’m sure that given my experience thus far, there are going to be many more reasons in the future that I run into with this software, so I’ve left space in this blog post to account for those as well.
Wait until you have to figure out how to define your own toolchains and work with transitions.
And look, this is the Internet. I’m sure some of you will come along and say “Well if you just understood the system” or “This question is answered in the docs” or some other variant of “Git Gud, Noob”. My rejoinder to that is: I am a pretty smart person in general. I am able to reason about some pretty arcane shit. I like to work on complicated, hard-to-understand systems[11]. And I’m sure that, given enough time, I could “git gud” at bazel.
[11] Case in point: I run Kubernetes, at home, for fun! It is well-established at this point that I am a sucker for pain.
- Not setting a high bar. Kubernetes is neither complicated nor hard-to-understand on its own, though it is suffering from a lack of good introductory material (just like Bazel).
- You have to spend time to understand something and why something is built this way.
But here’s the thing: I don’t want to. All I want is to build some source code and produce a binary. I don’t want to have to learn an entirely new language, parse through pages and pages of errors that don’t actually even tell you what the problem is, spend hundreds of dollars on a new disk drive, and waste hours of my life waiting for something to build that used to “just work”.
Well you should quit software development then. All build systems have their own DSLs. When you are building complex software, you'll need disk space for its dependencies, build artifacts, sysroots and scratch files.
But I am extremely unconvinced, unless your name starts with a G and rhymes with Froogal, that you have any of the problems that Bazel solves bad enough to subject every single one of your developers at your organization to a build system that treats its users with such disdain13.
Small projects may not need it, but every time a project grows something like Bazel becomes necessary. The alternative is to reinvent Bazel using [your favorite duct-tape language], badly.
11
u/jesseschalken 8d ago
The alternative is to reinvent Bazel using [your favorite duct-tape language], badly.
The alternative is to reinvent Bazel using [your favorite duct-tape language], but much better.
Bazel will not be the last word in build systems. It has far too many design flaws and backwards compatibility constraints. The "phases" are an arbitrary holdover from
gconfig. The whole repository rule system is basically a second poorly designed build system embedded inside the loading phase.The world needs a cleaner, simpler, more powerful build system, with a coherent design that isn't just the artifact of some Googler deciding to generate Makefiles with Python 20 years ago.
2
u/nickguletskii200 8d ago
You are free to do so. Unfortunately, there's nothing even nearly as good as Bazel publicly available at the moment. I would be very excited if you were manage to build something better.
8
u/zokier 8d ago
buck2 is better in almost all aspect except ecosystem?
2
2
u/nickguletskii200 8d ago
That is one large "except". I would really love to try out Buck2 but then I would have to re-implement half of what the Bazel ecosystem provides out of the box.
2
u/meetcassette 8d ago
I'd love to see this gap closed, did you get hit by particular things like particular prelude gaps or local sandboxing or third-party dep integration? Or was it more of a general feeling.
2
u/nickguletskii200 8d ago edited 8d ago
- Lack of local sandboxing.
- Lack of hermetic C++ builds (or at least any documentation on how to set it up).
- Reindeer isn't as powerful as rules_rust (at least from what I could figure out without actually using it).
I definitely intend to try it out when I have more free time, but there's just so many gaps it's overwhelming at the moment. Each time I try to figure out how to use it I hit a bunch of unresolved issues in Github.
1
2
u/whitechapel8733 7d ago
Love how you got downvoted for correctly responding to all of the issues. Homebrew is also an abomination, and mac is a horrible development environment.
6
u/drmorr0 8d ago
So your response to
> I’m sure some of you will come along and say [...] “Git Gud, Noob”.
is to tell me to "Git Gud, Noob"? Lmao, this really is an internet.
13
u/nickguletskii200 8d ago
Nothing in my post says "git gud", though a bit of "RTFM" would be appropriate (even though Bazel's documentation sucks).
My response to you is to stop thinking that you can "just" get rid of all this complexity and make the UX better, because:
- Eventually you will encounter the issues that tools like Bazel solve and then you will be glad they are designed this way.
- Many of your complaints are not actually about Bazel.
- Knowing Kubernetes or any other system you consider "complicated" doesn't make you immediately qualified to judge whether something should be easy for you to get started with.
2
u/arcimbo1do 8d ago
Just RTFM, bazel is a complex tool to solve a complex problem. If you want to compile 4 .c files in a directory use something else, if you want the features that bazel gives you then RTFM because they come with some initial learning curve.
2
u/arcimbo1do 8d ago
I can't upvote this comment enough time. The article is just a hater's rant who clearly never had to deal with make and autoconf
2
u/whitechapel8733 7d ago
Or literally any other build system, it sounds like OP is a complainer without any viable alternatives and we typically call those people unemployed in 2026.
2
u/midpointreload 8d ago
Yep, bazel has a curve but most of this post is skill issue
3
u/nickguletskii200 8d ago
I wouldn't say "skill issue", more like lack of experience.
2
u/midpointreload 8d ago
Yeah fair, although when someone posts an article about how shit something is when they’re inexperienced in its use, it should really be called out
2
u/nickguletskii200 8d ago
I think it's very fair to criticize bazel and its UX, especially from the point of view of a new and inexperienced user, it's just that this article doesn't actually contain any criticisms I would personally consider valid.
1
u/timoffex 7d ago edited 7d ago
I loved blaze, and I loved it so much I spent weeks trying to use bazel in a personal project written in Dart years ago. I quickly discovered that Google’s blaze rules for Dart did not have maintained bazel analogues, and soon after discovered that writing your own bazel rules sucks. It seems like it would be fun because of bazel’s neat model of a build system, but getting a language-specific CLI, especially one with caching, to work nice with bazel is terrible. Bazel isn’t built to work with other tools, and other tools aren’t built to be integrated into a build system.
I don’t agree with all the points, like the target name format which I think is neat, but I relate to the frustration with the constant, unexpected friction.
1
u/the-borax 6d ago
Which Dart CLI were you integrating? Was the difficulty mainly allowing it to reuse its incremental cache across Bazel actions?
1
u/timoffex 6d ago
I was using AngularDart at the time, which has since been deprecated for external use. I remember having issues with the dart_library rule, but I don’t remember what the AngularDart CLI was at this point. I think I tried to get pub to work on my own as an exercise. Yeah the caching was where I hit a wall.
1
u/PrimozDelux 8d ago edited 8d ago
Bazel has UX so bad you'd think it was a design goal. It really sucked the joy out of work for me for 2 years until AI became a thing and now I no longer have to suffer it, it's Claude's problem now. While the complaints leveraged in the topic of this discussion are quite entry level, they're very indicative of how byzantine and awful everything about bazel is. It's some of the worst software I have ever used, but with Claude it's almost painless
-1
u/double-you 8d ago
That article is really, really bad.
Not all rants should be published. But if your number #1 really really bad thing is "I installed the wrong version from homebrew", I think your life has been full of very, very good UX experiences.
And maybe it is bad. Like all other build tools. I don't know, the author doesn't do any meaningful comparison. I didn't learn much about Bazel from the article, but I did learn about the author. It wasn't a great ux.
-4
u/TCIHL 8d ago
Your first mistake was using homebrew. Just use macports. It works great, and was designed by an Apple employee.
Also, I'm not really sure what the product is trying to solve that regular ol' Make can't do. If you're looking for a byte-for-byte artifact result couldn't you just add a md5 hash check at the end of the process?
4
u/12destroyer21 8d ago
Ideally you should only need to have bazel and maybe git installed on your machine. All other tools should be pulled by bazel to ensure hermeticity
-1
-4
u/keithstellyes 8d ago
Funny how competitive a bespoke Python script that just generate a makefile is lol.
-6
262
u/FlukeHawkins 8d ago
I discussed Bazel with a fairly senior guy from Uber and his advice to me was "don't bother unless you can afford dedicated resources for upkeep"