Bazel Community Day – Munich¶

On October 23 2023, the day before the first European BazelCon, EngFlow and Tweag organized the sixth Bazel Community Day at the Salesforce office, capped off with a happy hour sponsored by Gradle.

The event kicked off early afternoon with snacks and welcome talks:

Introduction by Helen Altshuler, EngFlow and Andreas Herrmann, Tweag
Bazel at Salesforce by Gunnar Wagenknecht, Salesforce

After that, the event split into two parallel tracks:

Track 1:

A Bazel Beginner Bootcamp (by Billy Autrey, EngFlow)
Debugging Cache Misses and Sources of Nondeterminism in Bazel (by Ben Radford, Tweag and Joseph Gette (Mercedes)

The Bootcamp was delivered to a packed room with mixed experience levels, and received useful feedback which will be used to improve the next one.

"It was really good actually. I already use Bazel, but the Bootcamp helped me fill in some gaps."

The session on debugging cache misses was also in high demand. We had to move it to the main stage to provide enough room for all attendees. This session also received positive and constructive feedback.

Track 2:

Comprised a series of lightning talks:

Coverage with Bazel: An Overdue Summary, Ulf Adams from EngFlow
Bazel Migration Using Fully Ephemeral BUILD Files, Markus Hofbauer, Luminar
Fast Incremental Bazel Builds with ‘Persistent Pods’, Shishir Kumar, ThoughtSpot
Build Server Protocol, Andrzej Głuszak, Jetbrains
Buck2: Optimizations and Dynamic Dependencies, Neil Mitchell & Chris Hopman, Meta
Bazel + Go with rules_go and Gazelle, Tyler French & Zhongpeng Lin, Uber
Building JavaScrip & TypeSscript with rules_js, Greg Magolan, Aspect

There were also two unconference sessions. One focused on managing large monorepos and remote execution, while the other went into a deeper discussion on Buck2, Bazel, Starlark, and bzlmod. Bazel folks were interested to know more about Buck2's Starlark type checking, profiling, debugging, and dynamic actions.

We'll go into more detail about some of those talks and sessions. The recordings of the lightning talks are available on YouTube.

Intro and welcome¶

Helen Altshuler from EngFlow gave us a brief history of Bazel Community Day, a quick recap of the Build Meetup from a few days prior, and some numbers about Bazel uptake. She also mentioned that the next Community Day would be held in London, sometime in June 2024.

Andreas Herrmann from Tweag gave us a breakdown of the Community Day Structure and logistics for the day, and moderated the lightning talks.

Gunnar Wagenknecht from Salesforce gave us a detailed account of the scale of the Bazel migration at Salesforce. Many groups are using Bazel, some with up to 4000 engineers, multiple languages and operating systems, and up to 120000 Bazel targets. This was made possible by disallowing custom build rules, regulating load statements, and testing everything.

Coverage with Bazel: An Overdue Summary, Ulf Adams from EngFlow¶

After excitedly confessing to having had two colas before the talk, Ulf Adams sped into an overview of code coverage and Bazel.

Coverage keeps track of what code is executed

Ulf underscores that while low coverage may indicate subpar tests, a high coverage figure doesn't guarantee the quality of the tests. You also need to select tests based on coverage to target your changes to optimize resource utilization.

He went on to talk about how there is no right way to collect coverage, instead it can be done

per file
per class
per function
per line
per character

When looking at code coverage with Bazel, you find that different languages have different coverage stacks, each with it's own format! Before having a human look at the output you're probably going to need to post-process it:

With an output generator,
A report generator,
By manually generating the HTML for human consumption.

Bazel Migration Using Fully Ephemeral BUILD Files, Markus Hofbauer, Luminar¶

Markus gave us an account of how Luminar, a lidar system built from chip for autonomous driving, migrated to ephemeral BUILD files that aren't checked in to version control.

They were initially using Conan, a C++ package manager, to instrument CMake and split the monolithic CMake project into packages with interdependencies. That resulted in a handcrafted build system for the mono repo, which didn't work for them in the end.

Then they looked at automatically converting the Conan Buildfile into Bazel BUILD files, but

people introduced changes that broke the build
they didn't want to maintain two CI systems

So how did they solve this? By always auto-generating the Bazel BUILD files, using Gazelle, with the following constraints

always mapping 1 header file to 1 target, resulting in small targets
figuring out dependencies by looking at include files with naming conventions
handling edge cases with manual targets that override Gazelle
using NixOS for 3rd party dependencies
removing local BUILD files

All of which led them to a happy state with low Bazel maintenance so they could focus on developing new features.

Fast Incremental Bazel Builds with ‘Persistent Pods’, Shishir Kumar, ThoughtSpot¶

Shishir's opening line

"What do you want in life? Peace, long life, money, and faster builds"

got some laughs from the crowd. He went on to discuss the difference between clean and incremental builds, and how he leveraged warm Bazel caches with long living Kubernetes pods as Jenkins agents, and local disk caches as low-latency persistent volumes to achieve a 40% reduction in average Build and Test time.

Challenges included finding the right pod for the incoming build, minimizing the amount of work that you do, and syncing Git branches.

Ongoing work includes experimenting with different heuristics to minimize build time:

using the SHA from the previous build
diffing the targets between the two SHAs and using a smaller target list

All in all, they made good progress, but there are many avenues still left to explore.

Build Server Protocol, Andrzej Głuszak, Jetbrains¶

Andrzeij presented a lightning view of the Build Server Protocol (BSP), which creates an abstraction layer between editors and build tools, analogous to what the Language Server Protocol (LSP) did for language tools.

By using the Protocol, you can write once and run everywhere! Andrezej also mentioned a new IntelliJ plugin for Bazel that uses BSP.

Buck2: optimizations and dynamic dependencies, Neil Mitchell & Chris Hopman, Meta¶

Next up, Neil and Chris gave an account of the Buck2, the new primary build system at Meta, which is

Polyglot
For Monorepos
Open Source
Twice as fast as Buck1

Buck2 has target files in starlark specific to the project, rules which are all written in Starlark, a core binary written in Rust, providing the Starlark APIs, and a Starlark interpreter. It also has profiling, linting and a type checker built in.

The main reason to build Buck2 was to go faster, and as such, it has

a single dependency graph with no phases
remote execution, with pre-computed Merkle trees, and virtual files to provide builds without the bytes and inputs through EdenFS.

Some rules don't yet work, and it does not have a bzlmod-like package manager, but it does have dynamic dependencies the Buck Extension Language (BXL).

Learn more about the differences and similarities between Bazel and Buck2.

Bazel + Go with rules_go and Gazelle, Tyler French & Zhongpeng Lin, Uber¶

Tyler and Lin talked about how nogo, the rules_go static analysis tool, ran as part of the compile action, and how changing the nogo settings file invalidated the cache. An approach to solving this could be moving the nogo check out of the compile action.

They also talked about using Go modules with generated code, and handling dependencies with large MODULE.bazel files.

Building JavaScript & TypeScript with rules_js, Greg Magolan, Aspect¶

Greg started by describing the early state of JavaScript support in Bazel, which came out in 2017 with rules_nodejs but hit a dead end in terms of performance with third-party dependencies and runtime compatibility with JavaScript and TypeScript build tools. .

In 2021 Greg and Alex Eagle started writing rules_js, which was released in August 2022 and has gained steady adoption.

rules_js handles third-party dependencies much more efficiently, by working with pnpm lockfiles.

Debugging Cache Misses, Ben Radford - Tweag, Joseph Gette - Mercedes¶

Ben and Joseph co-presented a workshop on debugging cache misses.

It began with a brief explanation of hermeticity and determinism, which are important concepts to know about when trying to understand why a cache miss has occurred.

Then the audience were invited to clone the toy repository created for the workshop, and follow along as various issues were discovered, explained, and fixed.

Bonus: Games Night after BazelCon Day 1¶

The next day, in the evening after BazelCon Day 1, JetBrains hosted a games night at their Munich office, co-organized with EngFlow.

Spirits were high after a packed and exciting first conference day. Groups formed where people chatted with beverage in hand or played board games, card games, and chess.

We even got to enjoy a round of a Jeopardy-style quiz game about Bazel flags, featuring the new deck of Bazel flag playing cards EngFlow printed for the event. If you're interested in getting your hands on a deck, drop us a line.

Bringing Bazel Community Day to You¶

Fill in this survey if you’d like to join us at a future event and be informed about training and meetup opportunities.

Follow us on LinkedIn and X (Twitter), and join our newsletter to be the first to know about upcoming community days and other events!

join a future event

You can also read this article on the Tweag Blog.