Bazel Community Day in Amsterdam - Recap
Discover the highlights from the seventh Bazel Community Day and happy hour in Amsterdam. Jointly organized by EngFlow and Booking.com, hosted at the stunning new Amsterdam headquarters of Booking.com on March 25, 2024. You can see Booking.com's recap here.
The event started with an early afternoon of coffee and snacks, followed by a lineup of engaging talks:
- Opening remarks by Helen Altshuler
- "Adopting Bazel At Booking: A Bumpy Road" by Manuel Naranjo (Booking.com)
- "Reproducible Cloud-Based Dev Environments For Bazel" by Antonio Di Stefano and Jan Keromnes (EngFlow)
- "Bazel And Intellij 2024: State of IDE Integration" by Justin Kaeser (Jetbrains)
- "Lessons Learned: Adopting Bazel At Salesforce" by Gunnar Wagenknecht (Salesforce)
After the talks, the event branched into three concurrent "unconference" tracks, chosen based on the most popular topics suggested by attendees.
- Unconference #1 - Bazel IDE Integration and Developer Experience: A packed unconference all about the following: IDE integrations, shared sources, clean build graph, selling Bazel to users, and Starlark IDE support.
- Unconference # 2 - Remote Execution v3 and Virtual File System
- Unconference # 3 - Cross-compilation with Bazel
Opening Remarks
To kick things off, Helen Altshuler, co-founder and CEO of EngFlow, opened with a brief history of Bazel events. She shared the evolution of Bazel Community Days and Build Meetups and teased some upcoming events.
Her presentation went on to give an overview of EngFlow's history, including our remote build execution platform and the launch of our new self-host-able Kubernetes offering: my.engflow.com.
Adopting Bazel At Booking: A Bumpy Road by Manuel
After a warm welcome to Booking.com's new campus (which opened in May 2023), Manuel Naranjo, Senior Developer at Booking.com gave a great talk about their history and navigating Bazel. Since its inception in 1996, Booking.com has gone through a variety of technology migrations (ASP, to PHP, to Perl, to Java) to its most recent setup -- a large "distributed monorepo" using Bazel.
Manuel shared the challenges and triumphs of being one of Bazel's early adopters, including writing custom rules and impressive test times. He shared their evolving approach towards build infrastructure from internally creating everything to now wanting to offload their entire Bazel build backend to a trusted third-party.
RBE is very very hard... I don't want to do this myself anymore. Especially when you have thousands and thousands of files in your runfiles. We have 60,000 to 70,000 files on a single test, and Bazel just chokes on it.
Manuel's Bazel "war stories," hard-learned performance insights, and his outstanding memes made for one insightful and entertaining talk.
Reproducible cloud-based dev environments for Bazel by Antonio and Jan from EngFlow
EngFlow's Antonio Di Stefano and Jan Keromnes took the stage, walking us through an experiment using Cloud Development Environments (CDEs) combined with Remote Build Execution (RBE). Using the Envoy open source project as an example, the pair demonstrated the potential for the most optimal, quick developer experience with Bazel.
After introducing both tools - CDEs and RBE - they outlined the benefits of each. Using CDEs streamlines onboarding by creating a consistent environment for all developers. RBE gives users the quickest, most heavily-cached and parallel builds and tests possible. There are also unique benefits to combining the two, such as a centralized CI and build infrastructure co-located on protected, high-performance, and auto-scaling cloud servers.
In a live demo, Jan and Antonio demonstrated the power of using CDEs and RBE together. They fired up a complete, ready-to-go development environment for Envoy in a matter of seconds, a task that would take hours or even days without a CDE. Using this environment, they went on to demo a fully uncached build in about 2 minutes, a fraction of the time it'd take without RBE.
In the Q&A session, an audience member posed an interesting dilemma: How do CDEs work in poor network conditions? Surprisingly, CDEs perform better in degraded network conditions because they'll still quickly download large assets. This is because they leverage the speed of the data center between the environment and remote resources, allowing for high-bandwidth functions. The client-environment connection only needs web-app-like interactivity to issue remote commands.
Bazel + IntelliJ 2024 State of IDE integration by Justin from Jetbrains
Next, Justin Kaeser from JetBrains gave an overview of the current state of IntelliJ's Bazel support. After giving a bit of history on JetBrains and Bazel, Justin shared how they joined forces with Google to offer better IDE support for Bazel.
This collaboration with Google led to the creation of a new open source Bazel plugin by Justin's team at JetBrains. During his presentation, Justin shared the progress of the plugin, gave a demo of its UX, and showed off the latest features.
Lessons learned: Adopting Bazel at Salesforce by Gunnar from Salesforce
Gunnar Wagenknecht from Salesforce shared the company's ongoing transition to Bazel, highlighting both the challenges and successes encountered along the way. With around 3,500 developers, the migration proved to be complicated and time-consuming. Each step of the process took years, complicated by unexpected issues such as discovering an entire team relied on outdated software.
Despite the difficulties, Gunnar noted that support for the migration was easy to find and readily available. Salesforce used custom Maven plugins to measure build times and initial tests using Bazel showed a significant boost in performance. Some builds that took over 30 minutes were completed in just 4 minutes (P80). A dramatic improvement that made a compelling case for management to enforce a multi-year transition to Bazel.
It all started in 2018 and it's still ongoing! A journey that emphasizes the importance of a central vision embraced throughout the company. The challenges faced aren't only technical; a successful migration needs great culture and great people. As an example, Salesforce created a "Bazel champions" program that helps improve internal support and advocacy for the migration.
"The important piece I think you need to rethink in these large migration projects is [...] the vision. During all these years, it requires a lot of things to think about, but you always have to keep focused on the outcome. And the outcome, from my perspective, wasn't just Bazel. The outcome actually was much simpler: We need to kill Maven and Ant. That's the definition of done -- you're not done when you've migrated to Bazel. You're done when you've killed Ant and Maven."
Unconferences
Unconference 1: Bazel IDE Integration and Developer Experience
In the largest unconference breakout session, participants dived into a variety of topics including: - IDE integrations - Shared sources - Clean build graphs - Selling Bazel to users - Starlark IDE support (for developers writing Bazel rules)
This unconference focused on developer experience problems, specifically in today's Bazel IDE experience. One of the first issues discussed was the lack of support for dependency pinning in Java, with Gradle offering minimal functionality and Maven completely lacking support. Here are a few of the highlights from this unconference:
Using Custom Code With IDE Integration
Members of the JetBrains team noted that Bazel's extensive flexibility makes IDE integration challenging, especially for enterprises that have unique internal rules. Importing projects can lead to issues like unlocatable custom rules or providers, as well as optimizations that unintentionally break the IDE experience. JetBrains is considering addressing these issues by crafting extension points at the Aspect level, allowing developers to write custom code to improve their IDE support. This approach, unfortunately, needs a functional build, which may prove difficult in situations where IDE support is critical, such as refactorings, migrations, or debugging.
UX Issue With IDE Syncs
Scala developers from Databricks shared an issue with the UX where the IDE syncs successfully with the Google plugin, performs code auto-completion, and generally seems to be functional, but doesn't work when trying a Bazel build due to missing dependencies. This problem happens because the IDE "forgets" about Bazel after it's finished syncing and another sync is needed. Developers from JetBrains acknowledge this concern and assured that the new plugin will address them when it's released.
IDE Notification For Macros
One shared wish was for the IDE to identify and notify developers when they are working within a macro or handling code that was generated by a macro. For instance, when using a Java test and library one macro may only work for tests but not in libraries, posing challenges when investigating. The suggested solution for IDE developers was to use the Bazel query output build, which contains useful information on macros.
Starlark IDE Support
Additionally, unconference participants discussed the need for better syntactic and semantic checks for Starlark. Fortunately, an open source Language Server Protocol (LSP) for Starlark already exists. It's developed by Meta using Rust. Broader community efforts are in progress to improve typing for Starlark.
Unconference 2: Remote Execution v3 and Virtual File System
Another of the unconference sessions looked at a recent proposal by Ed Schouten et al. that took a look at Bazel Remote Execution v3. The discussion reviewed key aspects of the proposal including:
- Generalization of execution and output streams
- Cryptography
- CAS references as first-class citizens
- Elimination of large CAS objects
- Unification of Directories and Trees
- Generalization of the Action Cache
- Phasing out of the ByteStream/LongRunning protocol
- Action graphs
Mark Zeren from Broadcom expressed interest in a "Remote Execution native" file format specifically for archiving CAS objects. The goal is to craft 'tarballs' of files that are backed by CAS objects, especially the ones that don't actually copy the objects. The file format should also be compatible and backed by existing file systems that support block-level deduplication (e.g., ZFS).
There was also an interest in having better support for integrating Nix cache and build directories seamlessly into the build process.
Unconference 3: Cross-compilation with Bazel
In the third breakout session, it evolved into a Q&A session and presentation led by Yannic Bohnenberger from EngFlow, an expert on cross-compilation with Bazel.
The discussion highlighted common struggles in building across platforms like Mac, Linux, and Raspberry Pi, especially with non-hermetic self-build toolchains. The interest focused on generic approaches to cross-compilation, with members pointing out limitations of existing GitHub resources like these: Hermetic CC toolchain and LLVM toolchain for Bazel.
Unconference members from Booking.com asked about cross-compilation setups, leading Yannic to explain how EngFlow handles different execution and target platforms and the specifics of using Remote Build Execution with Mac laptops for distributed builds. Another member shared an experience they had with the grailbio LLVM toolchain for building on MacOS with Go and Rust while targeting Linux, highlighting the challenges of finding and setting up effective toolchains.
A recurring theme of Unconference 3 was the performance impact of multiple transitions in Bazel, which can cause a significant increase in memory usage and the amount of actions needing to be executed. Another challenge discussed was handling multiple versions of libraries within Bazel, a topic discussed at past BazelCons.
This session ended with a request for more detailed examples on configuring cross-compilation for various platforms and languages in Bazel environments, with a hope to improve understanding and implementation of these processes.
Bringing Bazel Community Day to You
We're always looking for new places to explore and more people to share community days with. Where should the next Bazel Community Day be? Fill in this form if you'd like to attend future events and receive updates on training sessions or meetup opportunities.
Connect with us on LinkedIn, X (Twitter), and subscribe to our newsletter to stay updated on upcoming community days and other events!
