In the previous Bzlmod post, we covered writing your own
Bazel module extensions to adapt your own setup code for dependencies that
aren't Bzlmod compatible. However, there are other Bzlmod incompatibilities and
related breakages that module extensions alone can't fix, such as forbidden API
usage or Windows path length errors.
This post shows you how to patch your dependencies and covers several situations
where patching is the only solution. We'll describe how to create and apply
patches for your dependencies, if you can't wait for upstream fixes (or
contribute them yourself).
EngFlow has multiple entities around the world, one of which is in Colombia. This year we’ve chosen Colombia as the site for our 2025 Leadership Summit, and I added a few days and weekends to explore the country with our team members, see how they live and meet their families. It’s been an incredible experience visiting Medellin, El Retiro, Itagui, Cali and Bogota, and I’ve captured a few learnings from this trip in a blog.
So far, we've covered relatively easy Bzlmod fixes, how to hold runfiles and
pkg_tar correctly, and how to access canonical repo names when absolutely
necessary. Now we'll discuss replacing WORKSPACE statements with your own
module extensions. You can use them to wrap your project's setup macros, and to
adapt external repositories that aren't Bzlmod compatible to work with your
Bzlmodified project.
As we look forward to 2025, we are grateful to our employees, customers, and the broader build community for the continued investment in achieving our mission of making developers productive and happy to keep Engineers in Flow.
In 2024 EngFlow continued our multi-year track record of exponential growth. This year we onboarded some of the most complex and advanced engineering organizations across autonomous driving, e-commerce, SaaS, finance, and chip manufacturing, resulting in significant developer velocity and cloud cost savings for these teams.
EngFlow has become more than a platform provider - we are elevating the developer experience practice for our customers, continuously pushing the cost and performance innovation, passing the savings directly to our customers. We are connecting engineers working on similar technologies to share best practices and talent across the Bazel, Buck2, Chromium, AOSP, and CMake ecosystems. While doing that, we're enriching our customers with music and are making the planet a greener place (read more about this below)!
Between roughly 2006 to 2008, Google developed remote caching and execution
technologies to scale its massive monorepo based software development
operation. This platform included Forge, the remote caching and execution
platform, and Blaze, a tool for building large multilanguage software projects,
eventually open sourced as Bazel. The advantages of this original platform
were so obvious that it literally sold itself, and ultimately inspired the
EngFlow platform.
Today, EngFlow is one of several competing remote caching and execution products
now available in the commercial space. This post describes how we continuously
benchmark our own product against different configurations to ensure that we
offer the best possible value. We hope that sharing our methodology might help
you evaluate whether remote caching or execution is right for your organization.
Bazel was announced on March 24, 2015, by unveiling https://bazel.io
and dumping code on https://github.com/bazelbuild/bazel. Its development has
been somewhat public ever since, but it has never been documented how
we got to that point. I was one of the people who started this
project, so let me tell you how it happened.
Co-hosted by EngFlow and Google, the build community in Tokyo came together for an afternoon of tech talks and a happy hour filled with beverages and good vibes.
The event went straight into talks, starting with introductory words from our Google host, Philipp and EngFlow Developer Support Engineer Kip. The talks that followed spanned build systems and build issues:
The previous two posts in this series showed how to use runfiles mechanisms
and rules_pkg mechanisms to avoid dealing with canonical repository names
under Bzlmod. However, one special case remains: when you need to depend on the
name of a repository directory, either at build time or runtime. This post
explains how to access canonical repository names in a portable way to solve
such problems. We'll use a macro when we can, and a custom Make
Variable when we can't, including when dealing with alias targets.
The Sydney to New York route was strategically scheduled via San Francisco to connect with our customers, employees and future customers there—and do some watercolors!
Fire up your time machine and set it to July 18, 2024, because we're taking a trip back to the CMake & Bazel Meetup in Munich. Let's set the scene: EngFlow and Apex.AI have joined forces to host a day filled with insightful talks and networking opportunities at Apex.AI's office in Munich.
Helen Altshuler & Dhruv Chad Kicking off the CMake & Bazel Meetup in Munich
Kicking off the evening, Dhruv from Apex.AI set the stage with a warm welcome. He shared his experience migrating Apex.AI to Bazel, touching on the challenges they faced and how this transition is shaping their work with ROS(Robot Operating System). This migration marks a big change in Apex.AI's development processes and how they interact with the ROS ecosystem. It was clear that Apex.AI isn't just adapting to the future; they're actively building it. Dhruv's talk set the tone for the evening, leaving us all eager to dive deeper into the world of modern build systems.
Evgeny Petrov from Quello took us on a wild ride. As the developer, producer, and operator of charging stations (talk about electrifying work), Evgeny presented his team's adventures in using Bazel for Rust embedded development. He outlined the hurdles they faced: dealing with third-party dependencies in Yocto images, the complexities of building system images in a monorepo, struggles of making everything work together in a monorepo, and the dance of cross-compilation configurations.
He shared their clever approach to solving these problems: a step dedicated to building Yocto images that's only triggered upon request, using a C++ toolchain not built by Yocto (due to integration challenges), and configuring Bazel for cross-compilation. His talk was a testament that with determination, even the most complex build system puzzles can be solved.
Nico Morin presenting at the CMake & Bazel Meetup in Munich
Nico Morin took the stage and shared how Apex.AI got Bazel and CMake to play nice. He began by outlining the challenges faced: despite their migration to Bazel, Apex.AI still needed to support CMake to meet customer expectations and maintain compatibility with ROS2 — without impacting maintenance efforts. So, how did they pull off this magic trick?
They developed an automated system to bridge the gap. By running Bazel aspects across their codebase, they collected the necessary build details — such as inputs and outputs for binaries and libraries. Using this information, they programmatically generated CMake files that mirrored their Bazel configurations. By implementing this automation it allows them to maintain Bazel as the single source of truth while still supporting CMake for customer-facing needs.
The result? A hybrid system where customers can choose to work with either CMake or Bazel, offering flexibility without sacrificing developer productivity or sanity.
Next, Damien from Tipi stepped up to introduce Common Package Specification (CPS). He presented us with an in-depth look. And he didn't stop there, he even demoed how they've implemented it.
CPS is a community effort to bring CMake Package Config files into a declarative format. With the potential to replace CMake Package Config in the future, CPS aims to fix how dependencies will be resolved. Best of all? It's not just for CMake — Bazel could use it too.
Damien gave a behind-the-scenes look at how to implement CPS on top of CMake's package configuration files. This approach allows for unheard-of optimization, building dependent code while dependencies are in the process of building, and only synchronizing at the link phase.
Michael Krasnyk presenting at the CMake & Bazel Meetup in Munich
To wrap up the lightning talks, Michael Krasnyk from Ruumi took us on an epic journey through the intricacies of Python setups across multiple versions and target platforms, reminiscent of Frodo's trek to Mordor. His presentation addressed the challenges and solutions for managing Python deployments in complex environments.
Michael showed how he leverages Bazel with a bazel run \<target> command for deployment automation, a unified pyproject.toml file to manage dependencies, and Poetry lock files to ensure consistency across multiple platforms and Python versions. By using the pyproject.toml file, Michael's solution centralizes dependency management, including platform-specific dependencies for AWS Lambda and EC2. Poetry handles version resolution and lock files, ensuring dependencies will be correctly managed across different Python environments.
This configuration works with Bazel custom rules, such as poetry_update for resolving dependencies and py_zip for packaging Python code into ZIP files tailored for AWS Lambda deployments. This approach resulted in simpler deployment processes across AWS services by automating package creation and performing compatibility checks with the Lambda execution environment. Michael also shared an integration with PDM (Python Development Master) and UV (for virtual environments) to further streamline dependency management.
Throughout his quest, Michael slayed some dragons — from running unit test on the host machine to integrating rules_oci for containerized workflows in deployment pipelines. He also tackled the complexities of Windows environment support, managing binary dependencies, and ensuring correct versioning of packages like shapely while excluding runtime dependencies from Lambda such as awslambdaric and boto3 to avoid redundancy in the packaged artifacts.
We're left with heads full of ideas and overflowing with excitement for what the future of build systems has in store. From embedded Bazel adventures to CMake — Bazel translators, from the future of CPS to TOML-powered Python wizardry, it's clear the build community is as innovative as ever.
Interested in attending or speaking at an event? We have much more coming up, so check it all out at meetup.build.
