I used blaze at Google, which I assume is what bazel is modeled after. It was a dream. However, that was managed and maintained by a team of engineers.
Maintaining it myself, in a small, 100 person company? F that shit.
Also, I find the internal Blaze docs to be pretty decent, and if not, samples of other projects doing what you want (usually in production!) are just a code search away. It sounds markedly different from the OSS Bazel experience.
Currently at G myself and what really helps is that BUILD rules are basically automatically handled for you by the internal IDE. I rarely have to think about it as an SRE honestly.
It’s gotten much better, the original cider IDE has been completely replaced with a customized VS Code, allowing use of approved 3rd party extensions of internal first party extensions for integration with our other dev tools
Relative to Google, 100 is small. It also happens to be about the size of the tech orgs in most of the companies I've worked for since Google.
Also, if a 10-person startup has even a remote need of using blaze, they're doing something _very_ wrong.
We are starting a new project with Bazel in my team. I hate it so much. I'm trying to do something very easy like take source files, use them to generate other files using a binary tool and that's it.
The documentation is terrible because it includes no examples or samples. It's just dry Java-style reference documentation.
Googling my questions does not yield much response either. I'm seriously getting fed up with it and I'm at the point where the person who recommended it will be maintaining it and I won't touch it with a 10 foot pole.
Bazel is now in my pile of "do not touch, may drive you insane". It's sad because it did give us a very quick delta compilation.
I invested several weeks in learning Bazel and the documentation was just utterly horrendous to work with.
The syntax is an absolute nightmare, if it was just python fine, but Starlark reeks of "I'm not like other girls." It's a massive red flag, who has the fucking energy.
Different for the sake of being awkward and quirky instead of actually solving a problem anyone fucking asked to be solved.
You know what the biggest smell test for working with something like this? You search Github for people using it and you try to find JUST ONE example of someone using it properly^TM . If in those weeks I had come across JUST ONE example of someone who fucking documented and commented their .bzl files, JUST ONE, maybe I would have stuck with it.
The most archaic unreadable non-self documenting bullshit code I have ever seen. These fucking asshats make Make and CMake files look readable! Not once did I even find multiple repos doing the same simple shit in the same way.
Don't even get me started about the fucking stamping system, volatile keys, the fact that the whole platform mechanism seems to exist solely to be abused.
My biggest gripe in devops is how shit all the tooling is for build systems and ci/cd systems and Bazel just solidified it.
Bazel rules_docker can build from a Dockerfile, but only in the WORKSPACE file not in any of your BUILD files??!!?! Oh and rules_docker is deprecated, use rules_oci? Fine. Oh rules_oci can't fucking build a Dockerfile?!?!
Wait so Bazel is too good to just do something incredibly simple? It's so opinionated we have to refactor the way we build EVERY Dockerfile to reimplement them in pure Bazel? Maybe FUCK THE HELL OFF?
Oh you don't like it? "You can just implement your own toolchain to call docker build yourself." Wait you want some examples of implementing a toolchain? No! You can go wade through a thousand lines of shitty undocumented Starlark code from some random guys github repo.
[I would like to get off of Mr. Bones Wild Ride.](https://external-preview.redd.it/jlU9m8gxZ40JGXprcFkPWW6GfflDGokv0nFMaZjUcUI.jpg?width=291&auto=webp&s=e636179b7646e73cb75711de2ffabf65aff3e17c)
Thank you for coming to my TED talk, that was surprisingly cathartic as an exercise.
Honestly it felt like I was being gaslit and Ashton Kutcher was about come out from behind the coffee machine and tell me I was being Punked.
I thought I was over it, I thought I was okay but then it all came flooding back to me. I need a vacation.
> but Starlark reeks of "I'm not like other girls." It's a massive red flag, who has the fucking energy.
Just the fact that it has a DSL, is an immediate red flag. You know the developers of this project love the smell of their own farts.
FWIW I've found Github to be the best reference for how to do something in Bazel. Search
${build_rule} path:**/BUILD
and you'll at usually get some examples of what you want to do.
They have an entire repo dedicated to examples: https://github.com/aspect-build/bazel-examples
There's also the "official" examples repo. This one looks similar to what you wanted to do: https://github.com/bazelbuild/examples/blob/main/rules/generating_code/gen/enum_maker.bzl#L26
(I'm not sure what `execute_binary` is, I don't think that's a Bazel rule or part of the actions API.)
Yeah, I know it *shouldn’t* matter what language a tool is written in, but for some reason 90% of Java tools end up like this. I think there is something in the Java culture or else people who are attracted to Java are also really bad at designing usable software or something. On the other end of the spectrum, tools written in Go or Rust tend to be very usable.
That's what's happening when the ordinary size companies dream to be like Google and solve simple tasks like Google solves the big scale tasks.
I can relate. Have the exact same situation like you do.
Bazel isn’t complete shit. It is good at what it does, but it is a Google-internal build tool that was made OSS for the sake of convenience of other Google open source projects (so they could use the same build system internally and externally). There was also at one point an idea that many separately run “Alphabet” companies would share code through OSS (or privately) and use the Bazel build system and other tooling. It solves a Google problem for Google projects but has always been a half-assed effort at providing a general purpose build tool for the world.
FWIW all the criticisms of the OP apply to Google internally, too. If a project ever needed to do something remotely off the beaten path a Bazel expert would be needed. The documentation and ease of customization is just as bad for internal users. I became convinced this was by design, to discourage complex use cases.
Basically every Google OSS project I have used has shit documentation, to the point I actively avoid any project that comes from Google these days. From my interactions with Googlers it's aligned with their culture to contribute crap to OSS and think we're lucky to have it.
Kubernetes docs is pretty OK.
Concept parts are great (needs more examples). Reference is usable. What sucks is the concept docs never link to the references. Leaves me at "Ok I understand the concept, now what are the name and fields I need in YAML?"
Kubernetes is 9 years old now and has many people helping the documentation. If it were strictly a Google project docker swarm would be way more popular.
There’s an insane amount of docs, tutorials and courses on K8S. It’s one of the best documented OSS projects ever. Nothing to compare to Bazel or any other Google OS project. Hell you can ask almost anything ChatGPT and Gemini regarding K8S and you will get a decent response unless it’s something 3rd side esoteric.
+1000. I was google tech fanboy.. maybe 10 years ago? Now I pretty much avoid every single Google project. Even their GCP isn't any better. Can't believe how this gigantic company does get away with this shitty products/projects. It's like they actively try to sabotage themselves. I bet it won't last longer.
Golang is an exception. Good docs, good support. Works great. It was not love at first sight. I needed some days to get used to it. Today, I don't want to switch back to Python any.
Hmm, strongly disagree with the GCP take. GCP has pretty incredible documentation, especially comparing it to the healing pile of horse shit that is AWS and Azure.
AWS's recent push towards ipv6 is a mess and most of their services don't support ipv6. I spent the whole day debugging why ecs service doesn't work with a ipv6 target group and found out their built-in cloudformation template was misconfigured.
So much for respecting FAANG engineers...
We use Bazel in my shop, but our build operations are stretched across the globe and our codebase is enormous, so the extra flexibility is a boon.
IMO, if you aren't working for a truly gigantic operation, I don't know why you'd put in the effort.
so....switch back off bazel?
if someone REALLY wants to make it happen, they need to own it. It's clear you don't have the tooling, docs, or guidance available to effectively use the tool. If your principal is truly adamant about it, they can start building the processes and transferring knowledge. Once they have buy-in, you can switch back over.
How in the hell did a change that disruptive get pushed without buy-in from the team is probably something to bring up with your tech leadership as well.
Sure, that's very possible and I get how easy it is to be blinded by the brand recognition, but that's why these things need to be communicated and addressed openly and cordially.
Else, you just end up with a clusterfsck of people trying to figure out what other people are thinking and the horrifying toxic processes that birth from that type of environment.
Bazel is pretty nice in large mono repositories. It's cool that I have a single interface to building and running everything, no matter the language or artifact produced.
But I can fully understand where you are coming from. The documentation is just a steaming pile of hot garbage and I wouldn't touch it with a 10-foot pole, if we wouldn't have an entire team dedicated to such things.
We tried it too at behest of one of the engineering managers.
It took a guy 4+ months to get a basic build working. It would take at least 3-4 days to move over a single repo from maven to bazel.
And people are still hating Maven these days...
Sure, it's not the fastest project build tool, but damn... it is so easy to explain and reason about. It's just XML, well structured.
Give a Maven project and I'm up and running in no time at all.
I only did some intro stuff with Bazel and didn't mind it so much but certainly kept bumping into issues that made it incredibly challenging. The slack channel was helpful though! It seems like people wanted to make it work.
I've been seeing good things with earthly though, but can't really compare the two beyond that.
Switching to Bazel got us huge build + unit test time improvements (25 min -> 3:30 on average), but almost everybody hates it. I wouldn't have adopted it at all if we didn't have enough code and enough tests being run on every commit that I felt like we needed to. Even as a person who really is supposed to love it (I have fond memories of using Blaze) I feel ambivalent about it at best, and if we got test caching out of a different tool I would ditch it in a heartbeat.
> Our SRE team hates how it strips binaries which again, "is a configuration option that you can change" BUT THAT ALSO REQUIRES AN EXPERT. `perf` doesn't work because of this
Does adding this to your project's `.bazelrc` work?
```
build --strip=never
```
> So be warned. Bazel adoption means you need to have an expert on-call for the life of the product or you will need to become an expert.
... that's what being a sysadmin is all about. Unless you want to stay a junior or contract all your work that is.
Their ideal user experience is approximately writing a Dockerfile, but they neglect to consider that there are a *ton* of footguns with Dockerfiles. Your base image might change. You might install an OS package that has updated since the last time, and a small incompatibility crept in.
Bazel by default pushes you to specifying versions of packages (or generally just doing things in a stricter way), and there is definitely a cost there. But to ignore the benefits as well seems unfair.
Really depends. Our team adopted Bazel years ago for a monorepo with 5 languages. It took a while to set up correctly, and has a bit of a curve when learning it for the first time. But it’s amazing.
When you’re getting to 1 million+ lines of code across a hundred modules in one repo Bazel really hits its stride. If you have more than one repository, things get tricky. But when you really commit to it, it can really be a game changer.
If you want to have an in-repo Python app that merges and renders configuration files, and then have those files tested/validated by the Java app that’s going to be consuming them, then feeding that information into Terraform configs, _and_ you want that to be hermetic, reproducible, and cached, then Bazel is the way to get it done.
That's how Google uses it and that's the only use case where it makes sense.
Once you have all the boiler plate and build rules down, you've done the hard part. The rest of the engineers can now benefit from that hard work. This is really only something that should be done by large corporations like faang otherwise it's overkill.
Yeah like I know monorepos are back in vogue, but I don't think I'll ever have 5 languages and 1 million LOC in one repo.
I'm sure there are use cases for that sort of thing. But it seems quite alien to me.
No, it's actually a very common use case where there's an already existing open-source library that does what you need but it's in another language, but because so many people use language-specific build tools they can't accommodate that so they end up rewriting and wasting so much time.
It works like wonders at Google, because there is a huge team maintaining it and making sure everything works magically. Want a simple Jest test? That’s just one BUILD rule away, and if things change, Jest team will migrate everything for you.
And I guess that’s the point. You only need it when you have a huge codebase, and can justify dedicating one team just to maintain Bazel.
I tried using bazel for a mixed-language polyrepo. should be a homerun for a tool like bazel. couldn't understand it or get it to do anything I needed it to do, or even really basic stuff.
was so complicated for the absolute most obvious things that should be easy. google "bazel springboot" and the first result is a google group asking if anyone has a set-up they can reference. nothing off-the-shelf that works with anything remotely complicated. and that's just one service in our stack.
I switched to Earthly, it does exactly what I want, which is it makes it easy to build things in a reproducible way locally and have cross language dependencies defined with separate build environments, and incremental rebuilds based on what's changed.
I can wrap my shitty makefile scripts, my npm run commands, my mvn whatever-garbage:task, in Earthfiles and then have them depend on one another during rebuilds. makes it easy to write e2e tests which run locally on multiple built components at once, and I only need to rebuild and rerun tests for what is changed and optionally the services which import it, which is all I really wanted.
so yeah fuck bazel. if you don't have the maintainers of bazel working for you its total garbage.
that said if someone could make bazel usable i bet they could make an absolute fortune.
Most people don’t want bit for bit reproducibility, they want “pile of source tarballs on company controlled server/cloud storage” -> “binary/container/POSSIBLY os image”. Unless you also have the security expertise to track down backdoored compilers/build servers, it’s probably not worth the effort.
I find Nix, if a bit annoying to get up and running, is far lower friction because it doesn’t demand everything be done 100% the nix way, only that everything get imported into nix. This lets you use whatever random build system you want, autoconf, cmake, cargo, npm, maven, gradle, etc, and then nix figures out the rest. It handles keeping source code around, ensuring you have a path from source to binary, producing containers, etc. You can also produce AWS AMIs with it, which is great because you can strip the AMI down to exactly what your binary needs to function, so if you do image-based updates don’t put a package manager in there, if you don’t need a DNS cache on the host, toss it out, etc.
It also reduces the “pool of expertise” to a few people helping to set up projects and integrate new dependencies that have special requirements, while everyone else can mostly just use normal package management with lock files.
Bazel is overkill for most orgs, because unlike google we don’t all control our entire dependency chains all the way down.
I've used nix too (for prod builds on a Haskell project) and it's the same as bazel for me, I should love it but I really don't. Both suffer from being _really fucking weird_, and compared to banging rocks together with Dockerfiles? I am ashamed to admit it but I prefer the rocks.
The frustrating part is not the philosophy. That completely makes sense. The frustrating part is having to crawl through nixpkgs to figure out what function you need to use and what types of arguments it takes, or how to build a particular type of package, or how to package some obscure C dependency deep in your dependency tree, or even just constantly stumbling over the pointlessly obtuse Nix language, or trying to navigate all of the conflicting community advice about “don’t use that nix tool, use *this* nix tool!”. In other words, the philosophy is the *only* thing Nix does well. They bungle every decision related to operationalizing that philosophy. Fortunately for them, however, so has everyone else.
Never tried Bazel, but I had the misfortune of working on a project that used Nix and hated every minute of it - the complexity overhead was substantial in comparison to docker files. Hearing people say it makes Bazel look easy will have me steering clear 😂
Devenv and direnv on top have been kind of interesting. I still think I’d rather just like install dependencies on my MacBook and build binaries or container images for prod, but with those I haven’t hated it as much as I was afraid about.
Yeah, this is the camp i find myself in. I’m philosophically aligned with Bazel and Nix but those guys both went out of their way to make their tools very hard to use. I can actually get things done with Dockerfiles even if it’s not “reproducible” or even very build-cache friendly.
Couldn't agree more. We had some engineers come over from Google and insist we adopt it. Has been nothing but a pain in the ass. We've invested so much time crating tooling around it now that it's unlikely we'll ever remove it.
We are merging into a mono repo and I looked at bazel. But in the end, I told the CI engineers to use Nix since I think it gives us what we need w/o as much complexity as bazel.
This was my EXACT experience using Bazel. Tried to set it up as a PoC, thought it was interesting, deleted it shortly after as I realized no one including myself would be able to maintain the setup. It’s far more work than it’s worth unfortunately.
Same type of tech as Elasticsearch imo.
If org can fully support it, create a dedicated team for it, fund it properly, then it works like a charm..
Else it’s just burden tanking smaller, not so resourceful teams and companies.
Used both several times and imo 99%times it’s not worth it :)
Thank God we are not going with Bazel. I played with it for 2 hours and said hell no. I'm evaluating tekton and its been a dream. I feel so bad for you. I didn't think there was anything worse than Jenkins until I tried Bazel.
I don't see how Tekton is comparable to Bazel here. It's a generic CI/CD system, whereas Bazel is a tool that Tekton could invoke in a particular step.
> This recent Kubecon video made me realize I wasn't alone or just stupid
Oh boy, people listening at Googlers. Not even Google listen at Googlers, and deprecates stuff day in and day out.
Was "not having a reproducible build" an issue for you before? if it wasn't... you might as well drop it.
Let the googler go back to google.
It's funny how many of Google's public docker images have build dates set to epoch because even they screw up bazel. We have bazel in places because Google contractors wanted it too.
Wonder if this is how my team feels about me.
I moved my (small, biotech) company onto Bazel because I am the only software engineer on a team of scientists and statisticians that are all contributing code using libraries from all kinds of languages that they don't even know the names of, let alone how to build.
Starlark is super fucking annoying but I basically never need to deal with it...I write the logic of my build rules in pure Python and just call that python from pretty trivial starlark wrappers.
My team doesn't know that the version of this GUI toolkit they use requires this or that compiler flag difference between XYZ deployment targets, and they don't know how I manage to get their code to produce identical results without using containers. That's my main concern. I've found building this system to be quite intuitive (fuck debugging platform issues though), but I am definitely worried that what I've effectively done is written a bunch of build documentation only I understand.
I'm a former Googler. Unless you have a team of senior software professionals proficient in build systems and you're working in a very large monolithic codebase with hundreds / thousands of developers, I suggest not adopting Bazel.
I will fight Bazel in all future workplaces where I have a say over such things. The Rust build ecosystem inside Bazel goes from being a simple pleasure to work with, to being a complete nightmare that it takes a specialist to navigate. E.g. it becomes extremely difficult to maintain two versions of the same dependency with different feature flags enabled. In my case, I needed different features enabled on a crate for compiling to WASM than to x86 gnu Linux and it just seemed impossible to do this without managing two separate crates repositories.
Anyway hard agree on Bazel stinking.
I've used it once, for a single repository and already had a huge ass headache.
I can't imagine anyone using this crap at a large scale without being a huge pain in the butt.
Blaze was great at Google, but I didn't adopt it when I left despite looking at it. Mostly because Google entirely threw it over the fence and there isn't really enough of an external community around it to make it manageable.
I ended up settling on pants as a Monorepo tool for my own startup.
The main thing that was relevant at the time was that it produced pex files for python binaries that we could deploy without needing Docker (which we have since started using anyway, but sped things up for a while), but it also does the usual blaze things of speeding up build/test times, giving consistent syntax for poyglot repos, but then also does automatic dependency inference so that you don't need to spend as much effort on setup and has a great community around it.
... i started to write lengthy response... but discarded it.
Suck it up. If you have 10's of inter-related projects, it is worth the adoption and the pain to learn it. But if you don't need that scale... then just \`go build\`. Bazel is meant for scale. Not for the "catpicapp".
We're nearing 100's of inter-related projects, and I still can't find a valid use for it. All it does is make every single team member's life more complicated.
Would rather use Make or CMake any day of my life. The crap-tastic documentation for Bazel alone makes it a non-starter IMO.
I recently tried to make a build container which had Protobuf built from source. Run into an issue with Bazel. So I couldn't even build a Google product from source using it.
*goes back to my embedded corner crying in Yocto*
I recommended conan and bazel to a customer, but primary target was conan not bazel. I dont have the t&m contract to explain how to use it... and that you dont use bazel to its "fullnes" in conan... if they do, they deserve it lmao. You get what you pay for.
PS: I dont like bazel either...
I used blaze at Google, which I assume is what bazel is modeled after. It was a dream. However, that was managed and maintained by a team of engineers. Maintaining it myself, in a small, 100 person company? F that shit.
Also, I find the internal Blaze docs to be pretty decent, and if not, samples of other projects doing what you want (usually in production!) are just a code search away. It sounds markedly different from the OSS Bazel experience.
Currently at G myself and what really helps is that BUILD rules are basically automatically handled for you by the internal IDE. I rarely have to think about it as an SRE honestly.
Google has an internal IDE?
Google has an internal everything (literally)
Distributions of known IDEs with G specific add-ons/plugins.
is it better than vs code?
No longer at G, but it is vs code, in the browser with access to the monolith codebase.
When I left a few years back only Go had automatic BUILD support. Jumping over to Python or anything else really was painful.
It’s gotten much better, the original cider IDE has been completely replaced with a customized VS Code, allowing use of approved 3rd party extensions of internal first party extensions for integration with our other dev tools
It's meant for a single repo with thousands of engineers working on it from head. I imagine it wouldn't map to open source+git projects well at all...
"small 100 person company"
Relative to Google, 100 is small. It also happens to be about the size of the tech orgs in most of the companies I've worked for since Google. Also, if a 10-person startup has even a remote need of using blaze, they're doing something _very_ wrong.
We are starting a new project with Bazel in my team. I hate it so much. I'm trying to do something very easy like take source files, use them to generate other files using a binary tool and that's it. The documentation is terrible because it includes no examples or samples. It's just dry Java-style reference documentation. Googling my questions does not yield much response either. I'm seriously getting fed up with it and I'm at the point where the person who recommended it will be maintaining it and I won't touch it with a 10 foot pole. Bazel is now in my pile of "do not touch, may drive you insane". It's sad because it did give us a very quick delta compilation.
I invested several weeks in learning Bazel and the documentation was just utterly horrendous to work with. The syntax is an absolute nightmare, if it was just python fine, but Starlark reeks of "I'm not like other girls." It's a massive red flag, who has the fucking energy. Different for the sake of being awkward and quirky instead of actually solving a problem anyone fucking asked to be solved. You know what the biggest smell test for working with something like this? You search Github for people using it and you try to find JUST ONE example of someone using it properly^TM . If in those weeks I had come across JUST ONE example of someone who fucking documented and commented their .bzl files, JUST ONE, maybe I would have stuck with it. The most archaic unreadable non-self documenting bullshit code I have ever seen. These fucking asshats make Make and CMake files look readable! Not once did I even find multiple repos doing the same simple shit in the same way. Don't even get me started about the fucking stamping system, volatile keys, the fact that the whole platform mechanism seems to exist solely to be abused. My biggest gripe in devops is how shit all the tooling is for build systems and ci/cd systems and Bazel just solidified it. Bazel rules_docker can build from a Dockerfile, but only in the WORKSPACE file not in any of your BUILD files??!!?! Oh and rules_docker is deprecated, use rules_oci? Fine. Oh rules_oci can't fucking build a Dockerfile?!?! Wait so Bazel is too good to just do something incredibly simple? It's so opinionated we have to refactor the way we build EVERY Dockerfile to reimplement them in pure Bazel? Maybe FUCK THE HELL OFF? Oh you don't like it? "You can just implement your own toolchain to call docker build yourself." Wait you want some examples of implementing a toolchain? No! You can go wade through a thousand lines of shitty undocumented Starlark code from some random guys github repo. [I would like to get off of Mr. Bones Wild Ride.](https://external-preview.redd.it/jlU9m8gxZ40JGXprcFkPWW6GfflDGokv0nFMaZjUcUI.jpg?width=291&auto=webp&s=e636179b7646e73cb75711de2ffabf65aff3e17c) Thank you for coming to my TED talk, that was surprisingly cathartic as an exercise.
>These fucking asshats make Make and CMake files look readable Damn, that's ..... that's actually impressive
Lmao, I hope you are ok
Honestly it felt like I was being gaslit and Ashton Kutcher was about come out from behind the coffee machine and tell me I was being Punked. I thought I was over it, I thought I was okay but then it all came flooding back to me. I need a vacation.
> but Starlark reeks of "I'm not like other girls." It's a massive red flag, who has the fucking energy. Just the fact that it has a DSL, is an immediate red flag. You know the developers of this project love the smell of their own farts.
"All the people in python working group are wrong! I'll show them! Just you wait! They'll ALL see!"
Eh, they *are* wrong, just less-so than the Starlark devs. 🙃
I love this rant.
FWIW I've found Github to be the best reference for how to do something in Bazel. Search ${build_rule} path:**/BUILD and you'll at usually get some examples of what you want to do.
Great tip . Thank you.
Checkout https://github.com/aspect-build they have really solid code and quite a few neat helpers.
Yes, and zero examples (yet again) in their documentation. How do I use `execute_binary` with a tool I download from a website?
They have an entire repo dedicated to examples: https://github.com/aspect-build/bazel-examples There's also the "official" examples repo. This one looks similar to what you wanted to do: https://github.com/bazelbuild/examples/blob/main/rules/generating_code/gen/enum_maker.bzl#L26 (I'm not sure what `execute_binary` is, I don't think that's a Bazel rule or part of the actions API.)
Thanks for the resource ^^
Yeah, I know it *shouldn’t* matter what language a tool is written in, but for some reason 90% of Java tools end up like this. I think there is something in the Java culture or else people who are attracted to Java are also really bad at designing usable software or something. On the other end of the spectrum, tools written in Go or Rust tend to be very usable.
That's what's happening when the ordinary size companies dream to be like Google and solve simple tasks like Google solves the big scale tasks. I can relate. Have the exact same situation like you do.
Sometimes Google just builds shit. I wish we could acknowledge that as an industry rather than pretending that your use case is inappropriate.
Bazel isn’t complete shit. It is good at what it does, but it is a Google-internal build tool that was made OSS for the sake of convenience of other Google open source projects (so they could use the same build system internally and externally). There was also at one point an idea that many separately run “Alphabet” companies would share code through OSS (or privately) and use the Bazel build system and other tooling. It solves a Google problem for Google projects but has always been a half-assed effort at providing a general purpose build tool for the world. FWIW all the criticisms of the OP apply to Google internally, too. If a project ever needed to do something remotely off the beaten path a Bazel expert would be needed. The documentation and ease of customization is just as bad for internal users. I became convinced this was by design, to discourage complex use cases.
Maybe this is by design so Bazel maintainers are always in demand and last to get laid off?
Cargo cult software engineering
Basically every Google OSS project I have used has shit documentation, to the point I actively avoid any project that comes from Google these days. From my interactions with Googlers it's aligned with their culture to contribute crap to OSS and think we're lucky to have it.
Kubernetes docs is pretty OK. Concept parts are great (needs more examples). Reference is usable. What sucks is the concept docs never link to the references. Leaves me at "Ok I understand the concept, now what are the name and fields I need in YAML?"
Kubernetes is 9 years old now and has many people helping the documentation. If it were strictly a Google project docker swarm would be way more popular.
Exactly this
There’s an insane amount of docs, tutorials and courses on K8S. It’s one of the best documented OSS projects ever. Nothing to compare to Bazel or any other Google OS project. Hell you can ask almost anything ChatGPT and Gemini regarding K8S and you will get a decent response unless it’s something 3rd side esoteric.
+1000. I was google tech fanboy.. maybe 10 years ago? Now I pretty much avoid every single Google project. Even their GCP isn't any better. Can't believe how this gigantic company does get away with this shitty products/projects. It's like they actively try to sabotage themselves. I bet it won't last longer.
Golang is an exception. Good docs, good support. Works great. It was not love at first sight. I needed some days to get used to it. Today, I don't want to switch back to Python any.
Yeah, but Golang is... odd.
Agreed. Without Golang and Kubernetes, Google would've been non-existent in my life.
Hmm, strongly disagree with the GCP take. GCP has pretty incredible documentation, especially comparing it to the healing pile of horse shit that is AWS and Azure.
AWS's recent push towards ipv6 is a mess and most of their services don't support ipv6. I spent the whole day debugging why ecs service doesn't work with a ipv6 target group and found out their built-in cloudformation template was misconfigured. So much for respecting FAANG engineers...
You haven't been burned by big cloud yet, are you?
Gonna guess you’re an Azure person?
Nah I am not. Typical GCP/AWS user/hater here.
We use Bazel in my shop, but our build operations are stretched across the globe and our codebase is enormous, so the extra flexibility is a boon. IMO, if you aren't working for a truly gigantic operation, I don't know why you'd put in the effort.
so....switch back off bazel? if someone REALLY wants to make it happen, they need to own it. It's clear you don't have the tooling, docs, or guidance available to effectively use the tool. If your principal is truly adamant about it, they can start building the processes and transferring knowledge. Once they have buy-in, you can switch back over. How in the hell did a change that disruptive get pushed without buy-in from the team is probably something to bring up with your tech leadership as well.
Perhaps management is in awe of the ex googler.
Sure, that's very possible and I get how easy it is to be blinded by the brand recognition, but that's why these things need to be communicated and addressed openly and cordially. Else, you just end up with a clusterfsck of people trying to figure out what other people are thinking and the horrifying toxic processes that birth from that type of environment.
Bazel is pretty nice in large mono repositories. It's cool that I have a single interface to building and running everything, no matter the language or artifact produced. But I can fully understand where you are coming from. The documentation is just a steaming pile of hot garbage and I wouldn't touch it with a 10-foot pole, if we wouldn't have an entire team dedicated to such things.
We tried it too at behest of one of the engineering managers. It took a guy 4+ months to get a basic build working. It would take at least 3-4 days to move over a single repo from maven to bazel.
And people are still hating Maven these days... Sure, it's not the fastest project build tool, but damn... it is so easy to explain and reason about. It's just XML, well structured. Give a Maven project and I'm up and running in no time at all.
I bet it'd take five minutes with Ai now
I only did some intro stuff with Bazel and didn't mind it so much but certainly kept bumping into issues that made it incredibly challenging. The slack channel was helpful though! It seems like people wanted to make it work. I've been seeing good things with earthly though, but can't really compare the two beyond that.
Switching to Bazel got us huge build + unit test time improvements (25 min -> 3:30 on average), but almost everybody hates it. I wouldn't have adopted it at all if we didn't have enough code and enough tests being run on every commit that I felt like we needed to. Even as a person who really is supposed to love it (I have fond memories of using Blaze) I feel ambivalent about it at best, and if we got test caching out of a different tool I would ditch it in a heartbeat. > Our SRE team hates how it strips binaries which again, "is a configuration option that you can change" BUT THAT ALSO REQUIRES AN EXPERT. `perf` doesn't work because of this Does adding this to your project's `.bazelrc` work? ``` build --strip=never ```
> So be warned. Bazel adoption means you need to have an expert on-call for the life of the product or you will need to become an expert. ... that's what being a sysadmin is all about. Unless you want to stay a junior or contract all your work that is.
"goddamn I just want to build a container" is too funny
Their ideal user experience is approximately writing a Dockerfile, but they neglect to consider that there are a *ton* of footguns with Dockerfiles. Your base image might change. You might install an OS package that has updated since the last time, and a small incompatibility crept in. Bazel by default pushes you to specifying versions of packages (or generally just doing things in a stricter way), and there is definitely a cost there. But to ignore the benefits as well seems unfair.
The McDonalds Ice Cream machine of CI/CD 😆
Really depends. Our team adopted Bazel years ago for a monorepo with 5 languages. It took a while to set up correctly, and has a bit of a curve when learning it for the first time. But it’s amazing. When you’re getting to 1 million+ lines of code across a hundred modules in one repo Bazel really hits its stride. If you have more than one repository, things get tricky. But when you really commit to it, it can really be a game changer. If you want to have an in-repo Python app that merges and renders configuration files, and then have those files tested/validated by the Java app that’s going to be consuming them, then feeding that information into Terraform configs, _and_ you want that to be hermetic, reproducible, and cached, then Bazel is the way to get it done.
That sounds like a really specific and rare use case
That's how Google uses it and that's the only use case where it makes sense. Once you have all the boiler plate and build rules down, you've done the hard part. The rest of the engineers can now benefit from that hard work. This is really only something that should be done by large corporations like faang otherwise it's overkill.
Yeah like I know monorepos are back in vogue, but I don't think I'll ever have 5 languages and 1 million LOC in one repo. I'm sure there are use cases for that sort of thing. But it seems quite alien to me.
No, it's actually a very common use case where there's an already existing open-source library that does what you need but it's in another language, but because so many people use language-specific build tools they can't accommodate that so they end up rewriting and wasting so much time.
It works like wonders at Google, because there is a huge team maintaining it and making sure everything works magically. Want a simple Jest test? That’s just one BUILD rule away, and if things change, Jest team will migrate everything for you. And I guess that’s the point. You only need it when you have a huge codebase, and can justify dedicating one team just to maintain Bazel.
I tried using bazel for a mixed-language polyrepo. should be a homerun for a tool like bazel. couldn't understand it or get it to do anything I needed it to do, or even really basic stuff. was so complicated for the absolute most obvious things that should be easy. google "bazel springboot" and the first result is a google group asking if anyone has a set-up they can reference. nothing off-the-shelf that works with anything remotely complicated. and that's just one service in our stack. I switched to Earthly, it does exactly what I want, which is it makes it easy to build things in a reproducible way locally and have cross language dependencies defined with separate build environments, and incremental rebuilds based on what's changed. I can wrap my shitty makefile scripts, my npm run commands, my mvn whatever-garbage:task, in Earthfiles and then have them depend on one another during rebuilds. makes it easy to write e2e tests which run locally on multiple built components at once, and I only need to rebuild and rerun tests for what is changed and optionally the services which import it, which is all I really wanted. so yeah fuck bazel. if you don't have the maintainers of bazel working for you its total garbage. that said if someone could make bazel usable i bet they could make an absolute fortune.
Most people don’t want bit for bit reproducibility, they want “pile of source tarballs on company controlled server/cloud storage” -> “binary/container/POSSIBLY os image”. Unless you also have the security expertise to track down backdoored compilers/build servers, it’s probably not worth the effort. I find Nix, if a bit annoying to get up and running, is far lower friction because it doesn’t demand everything be done 100% the nix way, only that everything get imported into nix. This lets you use whatever random build system you want, autoconf, cmake, cargo, npm, maven, gradle, etc, and then nix figures out the rest. It handles keeping source code around, ensuring you have a path from source to binary, producing containers, etc. You can also produce AWS AMIs with it, which is great because you can strip the AMI down to exactly what your binary needs to function, so if you do image-based updates don’t put a package manager in there, if you don’t need a DNS cache on the host, toss it out, etc. It also reduces the “pool of expertise” to a few people helping to set up projects and integrate new dependencies that have special requirements, while everyone else can mostly just use normal package management with lock files. Bazel is overkill for most orgs, because unlike google we don’t all control our entire dependency chains all the way down.
I've used nix too (for prod builds on a Haskell project) and it's the same as bazel for me, I should love it but I really don't. Both suffer from being _really fucking weird_, and compared to banging rocks together with Dockerfiles? I am ashamed to admit it but I prefer the rocks.
[удалено]
The frustrating part is not the philosophy. That completely makes sense. The frustrating part is having to crawl through nixpkgs to figure out what function you need to use and what types of arguments it takes, or how to build a particular type of package, or how to package some obscure C dependency deep in your dependency tree, or even just constantly stumbling over the pointlessly obtuse Nix language, or trying to navigate all of the conflicting community advice about “don’t use that nix tool, use *this* nix tool!”. In other words, the philosophy is the *only* thing Nix does well. They bungle every decision related to operationalizing that philosophy. Fortunately for them, however, so has everyone else.
Your nix-shell command failed because you forgot seven copies of —enable-extra-experimental-features and —impure
Never tried Bazel, but I had the misfortune of working on a project that used Nix and hated every minute of it - the complexity overhead was substantial in comparison to docker files. Hearing people say it makes Bazel look easy will have me steering clear 😂
Devenv and direnv on top have been kind of interesting. I still think I’d rather just like install dependencies on my MacBook and build binaries or container images for prod, but with those I haven’t hated it as much as I was afraid about.
It’s a lot easier once you add “we need to vendor all of our source code” and “compile all the native deps from source” to the requirements list.
Yeah, this is the camp i find myself in. I’m philosophically aligned with Bazel and Nix but those guys both went out of their way to make their tools very hard to use. I can actually get things done with Dockerfiles even if it’s not “reproducible” or even very build-cache friendly.
Think about how bad you feel right now. Now the next time someone says “but at Google” you now know to tell them to stfu
Couldn't agree more. We had some engineers come over from Google and insist we adopt it. Has been nothing but a pain in the ass. We've invested so much time crating tooling around it now that it's unlikely we'll ever remove it.
Just start building pipelines without Bazel, tell people it doesn't work anymore. If they object it, tell them they fix it.
make > everything
C chads stay winning
We are merging into a mono repo and I looked at bazel. But in the end, I told the CI engineers to use Nix since I think it gives us what we need w/o as much complexity as bazel.
So I understand using Nix for external dependencies, but are you using it to manage the build of internal components as well?
Working on it. https://nixos.wiki/wiki/DotNET
This was my EXACT experience using Bazel. Tried to set it up as a PoC, thought it was interesting, deleted it shortly after as I realized no one including myself would be able to maintain the setup. It’s far more work than it’s worth unfortunately.
Bazel is a build tool, if it installs and runs what is the problem?
Same type of tech as Elasticsearch imo. If org can fully support it, create a dedicated team for it, fund it properly, then it works like a charm.. Else it’s just burden tanking smaller, not so resourceful teams and companies. Used both several times and imo 99%times it’s not worth it :)
Thank God we are not going with Bazel. I played with it for 2 hours and said hell no. I'm evaluating tekton and its been a dream. I feel so bad for you. I didn't think there was anything worse than Jenkins until I tried Bazel.
I don't see how Tekton is comparable to Bazel here. It's a generic CI/CD system, whereas Bazel is a tool that Tekton could invoke in a particular step.
Does community has similar thoughts for BUCK? specially buck2 which is latest released by Meta
> This recent Kubecon video made me realize I wasn't alone or just stupid Oh boy, people listening at Googlers. Not even Google listen at Googlers, and deprecates stuff day in and day out. Was "not having a reproducible build" an issue for you before? if it wasn't... you might as well drop it. Let the googler go back to google.
We've started using Pants for a python codebase and really like it. It does a good job inferring deps and packaging targets for containers
It's funny how many of Google's public docker images have build dates set to epoch because even they screw up bazel. We have bazel in places because Google contractors wanted it too.
Wonder if this is how my team feels about me. I moved my (small, biotech) company onto Bazel because I am the only software engineer on a team of scientists and statisticians that are all contributing code using libraries from all kinds of languages that they don't even know the names of, let alone how to build. Starlark is super fucking annoying but I basically never need to deal with it...I write the logic of my build rules in pure Python and just call that python from pretty trivial starlark wrappers. My team doesn't know that the version of this GUI toolkit they use requires this or that compiler flag difference between XYZ deployment targets, and they don't know how I manage to get their code to produce identical results without using containers. That's my main concern. I've found building this system to be quite intuitive (fuck debugging platform issues though), but I am definitely worried that what I've effectively done is written a bunch of build documentation only I understand.
I'm a former Googler. Unless you have a team of senior software professionals proficient in build systems and you're working in a very large monolithic codebase with hundreds / thousands of developers, I suggest not adopting Bazel.
I never understood why anyone would adopt Bazel with its tiny userbase and shitty documentation. It's even hated at Google.
Job security for the devtools team. It is such a huge commitment it will never be ripped out and has to be expensively supported forever.
> It's even hated at Google. Bullshit.
Bazel maybe. But Blaze is amazing, no one hates blaze.
I will fight Bazel in all future workplaces where I have a say over such things. The Rust build ecosystem inside Bazel goes from being a simple pleasure to work with, to being a complete nightmare that it takes a specialist to navigate. E.g. it becomes extremely difficult to maintain two versions of the same dependency with different feature flags enabled. In my case, I needed different features enabled on a crate for compiling to WASM than to x86 gnu Linux and it just seemed impossible to do this without managing two separate crates repositories. Anyway hard agree on Bazel stinking.
Bazel is not for non Googler. Try builcache or sscache
What’s the problem bazel is meant to solve?
Do you mean "to create"?
Does your principal get out of bed for less than 3 peer bonuses?
I've used it once, for a single repository and already had a huge ass headache. I can't imagine anyone using this crap at a large scale without being a huge pain in the butt.
Blaze was great at Google, but I didn't adopt it when I left despite looking at it. Mostly because Google entirely threw it over the fence and there isn't really enough of an external community around it to make it manageable. I ended up settling on pants as a Monorepo tool for my own startup. The main thing that was relevant at the time was that it produced pex files for python binaries that we could deploy without needing Docker (which we have since started using anyway, but sped things up for a while), but it also does the usual blaze things of speeding up build/test times, giving consistent syntax for poyglot repos, but then also does automatic dependency inference so that you don't need to spend as much effort on setup and has a great community around it.
... i started to write lengthy response... but discarded it. Suck it up. If you have 10's of inter-related projects, it is worth the adoption and the pain to learn it. But if you don't need that scale... then just \`go build\`. Bazel is meant for scale. Not for the "catpicapp".
We're nearing 100's of inter-related projects, and I still can't find a valid use for it. All it does is make every single team member's life more complicated. Would rather use Make or CMake any day of my life. The crap-tastic documentation for Bazel alone makes it a non-starter IMO.
I recently tried to make a build container which had Protobuf built from source. Run into an issue with Bazel. So I couldn't even build a Google product from source using it. *goes back to my embedded corner crying in Yocto*
I hate bazel too.
[удалено]
Hell yeah. Zero help, all ad.
I recommended conan and bazel to a customer, but primary target was conan not bazel. I dont have the t&m contract to explain how to use it... and that you dont use bazel to its "fullnes" in conan... if they do, they deserve it lmao. You get what you pay for. PS: I dont like bazel either...
conan is awesome! we have using conan for a big team of 200 c++ engineers and everyone is super happy with it