EngFlow 2024 Year End Wrap

As we look forward to 2025, we are grateful to our employees, customers, and the broader build community for the continued investment in achieving our mission of making developers productive and happy to keep Engineers in Flow.

Holiday baking

In 2024 EngFlow continued our multi-year track record of exponential growth. This year we onboarded some of the most complex and advanced engineering organizations across autonomous driving, e-commerce, SaaS, finance, and chip manufacturing, resulting in significant developer velocity and cloud cost savings for these teams.

EngFlow has become more than a platform provider - we are elevating the developer experience practice for our customers, continuously pushing the cost and performance innovation, passing the savings directly to our customers. We are connecting engineers working on similar technologies to share best practices and talent across the Bazel, Buck2, Chromium, AOSP, and CMake ecosystems. While doing that, we're enriching our customers with music and are making the planet a greener place (read more about this below)!

How to Evaluate Remote Caching and Execution

Between roughly 2006 to 2008, Google developed remote caching and execution technologies to scale its massive monorepo based software development operation. This platform included Forge, the remote caching and execution platform, and Blaze, a tool for building large multilanguage software projects, eventually open sourced as Bazel. The advantages of this original platform were so obvious that it literally sold itself, and ultimately inspired the EngFlow platform.

Today, EngFlow is one of several competing remote caching and execution products now available in the commercial space. This post describes how we continuously benchmark our own product against different configurations to ensure that we offer the best possible value. We hope that sharing our methodology might help you evaluate whether remote caching or execution is right for your organization.

Build Meetup in Tokyo — Recap

Co-hosted by EngFlow and Google, the build community in Tokyo came together for an afternoon of tech talks and a happy hour filled with beverages and good vibes.

The event went straight into talks, starting with introductory words from our Google host, Philipp and EngFlow Developer Support Engineer Kip. The talks that followed spanned build systems and build issues:

Following the talks, we headed out to a nearby bar for some canapés and drinks, meeting fellow build engineers and enthusiasts from around the region.

Migrating to Bazel Modules (a.k.a. Bzlmod) - Repo Names, Macros, and Variables

The previous two posts in this series showed how to use runfiles mechanisms and rules_pkg mechanisms to avoid dealing with canonical repository names under Bzlmod. However, one special case remains: when you need to depend on the name of a repository directory, either at build time or runtime. This post explains how to access canonical repository names in a portable way to solve such problems. We'll use a macro when we can, and a custom Make Variable when we can't, including when dealing with alias targets.

CMake & Bazel Meetup in Munich - Recap

Fire up your time machine and set it to July 18, 2024, because we're taking a trip back to the CMake & Bazel Meetup in Munich. Let's set the scene: EngFlow and Apex.AI have joined forces to host a day filled with insightful talks and networking opportunities at Apex.AI's office in Munich.

CMake & Bazel Meetup - Jul 2024 - Munich

The CMake & Bazel Meetup in Munich

There was no shortage of great talks and knowledge, including: - Welcome and Introduction by Dhruv Chad, Apex.AI - Deploying With Bazel on Yocto-based System by Evgeny Petrov, Quello - Automatically Translating Our Bazel Codebase to CMake by Nico Morin, Apex.AI - CMake CPS: What It Is and Why It Matters by Damien Buhl, Tipi - One TOML File to Rule Them All: Official Rules for Multi-Version Multi-Target Python Setup by Michael Krasnyk, Ruumi - Until Next Time

Welcome and Introduction by Dhruv Chad, Apex.AI

Kickoff Remarks

Helen Altshuler & Dhruv Chad Kicking off the CMake & Bazel Meetup in Munich

Kicking off the evening, Dhruv from Apex.AI set the stage with a warm welcome. He shared his experience migrating Apex.AI to Bazel, touching on the challenges they faced and how this transition is shaping their work with ROS(Robot Operating System). This migration marks a big change in Apex.AI's development processes and how they interact with the ROS ecosystem. It was clear that Apex.AI isn't just adapting to the future; they're actively building it. Dhruv's talk set the tone for the evening, leaving us all eager to dive deeper into the world of modern build systems.

Deploying With Bazel on Yocto-based System by Evgeny Petrov, Quello

Evgeny Petrov from Quello took us on a wild ride. As the developer, producer, and operator of charging stations (talk about electrifying work), Evgeny presented his team's adventures in using Bazel for Rust embedded development. He outlined the hurdles they faced: dealing with third-party dependencies in Yocto images, the complexities of building system images in a monorepo, struggles of making everything work together in a monorepo, and the dance of cross-compilation configurations.

He shared their clever approach to solving these problems: a step dedicated to building Yocto images that's only triggered upon request, using a C++ toolchain not built by Yocto (due to integration challenges), and configuring Bazel for cross-compilation. His talk was a testament that with determination, even the most complex build system puzzles can be solved.

Automatically Translating Our Bazel Codebase to CMake by Nico Morin, Apex.AI

See the slides

Automatically Translate Bazel Codebase to CMake

Nico Morin presenting at the CMake & Bazel Meetup in Munich

Nico Morin took the stage and shared how Apex.AI got Bazel and CMake to play nice. He began by outlining the challenges faced: despite their migration to Bazel, Apex.AI still needed to support CMake to meet customer expectations and maintain compatibility with ROS2 — without impacting maintenance efforts. So, how did they pull off this magic trick?

They developed an automated system to bridge the gap. By running Bazel aspects across their codebase, they collected the necessary build details — such as inputs and outputs for binaries and libraries. Using this information, they programmatically generated CMake files that mirrored their Bazel configurations. By implementing this automation it allows them to maintain Bazel as the single source of truth while still supporting CMake for customer-facing needs.

The result? A hybrid system where customers can choose to work with either CMake or Bazel, offering flexibility without sacrificing developer productivity or sanity.

CMake CPS: What It Is and Why It Matters by Damien Buhl, Tipi

Next, Damien from Tipi stepped up to introduce Common Package Specification (CPS). He presented us with an in-depth look. And he didn't stop there, he even demoed how they've implemented it.

CPS is a community effort to bring CMake Package Config files into a declarative format. With the potential to replace CMake Package Config in the future, CPS aims to fix how dependencies will be resolved. Best of all? It's not just for CMake — Bazel could use it too.

Damien gave a behind-the-scenes look at how to implement CPS on top of CMake's package configuration files. This approach allows for unheard-of optimization, building dependent code while dependencies are in the process of building, and only synchronizing at the link phase.

One TOML File to Rule Them All: Official Rules for Multi-Version Multi-Target Python Setup by Michael Krasnyk, Ruumi

See the slides

One TOML File to Rule Them All

Michael Krasnyk presenting at the CMake & Bazel Meetup in Munich

To wrap up the lightning talks, Michael Krasnyk from Ruumi took us on an epic journey through the intricacies of Python setups across multiple versions and target platforms, reminiscent of Frodo's trek to Mordor. His presentation addressed the challenges and solutions for managing Python deployments in complex environments.

Michael showed how he leverages Bazel with a bazel run \<target> command for deployment automation, a unified pyproject.toml file to manage dependencies, and Poetry lock files to ensure consistency across multiple platforms and Python versions. By using the pyproject.toml file, Michael's solution centralizes dependency management, including platform-specific dependencies for AWS Lambda and EC2. Poetry handles version resolution and lock files, ensuring dependencies will be correctly managed across different Python environments.

This configuration works with Bazel custom rules, such as poetry_update for resolving dependencies and py_zip for packaging Python code into ZIP files tailored for AWS Lambda deployments. This approach resulted in simpler deployment processes across AWS services by automating package creation and performing compatibility checks with the Lambda execution environment. Michael also shared an integration with PDM (Python Development Master) and UV (for virtual environments) to further streamline dependency management.

Throughout his quest, Michael slayed some dragons — from running unit test on the host machine to integrating rules_oci for containerized workflows in deployment pipelines. He also tackled the complexities of Windows environment support, managing binary dependencies, and ensuring correct versioning of packages like shapely while excluding runtime dependencies from Lambda such as awslambdaric and boto3 to avoid redundancy in the packaged artifacts.

Until Next Time

We're left with heads full of ideas and overflowing with excitement for what the future of build systems has in store. From embedded Bazel adventures to CMake — Bazel translators, from the future of CPS to TOML-powered Python wizardry, it's clear the build community is as innovative as ever.

Interested in attending or speaking at an event? We have much more coming up, so check it all out at meetup.build.

Migrating to Bazel Modules (a.k.a. Bzlmod) - Repo Names and rules_pkg

The previous post in our Bzlmod migration series demonstrated how to make runfiles paths portable to a Bzlmod world. Another common source of Bzlmod file path breakages are misconfigured rules from rules_pkg, which contains rules for building archives from build outputs and/or external repositories. This post will explain key details of some of these rules, so you can stop "holding it wrong" and easily migrate archive targets to Bzlmod.

Sydney: Around the World with Bazel in Watercolors

This article is part of the series "Around the World with Bazel in Watercolors".

As the images hint, I was mostly inspired by the rich Tasmanian landscape, which is where we went on a family vacation. I did, however, sandwich that week between working for a week in Sydney and a week in San Francisco. In Sydney I couldn’t pass on an opportunity to meet with our customers while visiting family, and to gather local engineers passionate about Bazel (all 10 of them! 😃) from Canva, Splunk, Snap, MongoDB, and more for a Bazel meetup.

Sydney Watercolors

Migrating to Bazel Modules (a.k.a. Bzlmod) - Repo Names and Runfiles

The first post in our Bzlmod migration series explained many of the problems that may arise when migrating your project. These next three posts will explore various solutions to problems arising from changes in how Bazel handles repository names under Bzlmod, beginning with runfiles paths. After applying the techniques in this post, your project should be well insulated from runfiles path related breakages, now and well into the future.