It's been almost six years since the previous entry in this series was originally published, and there are many new topics to discuss!
The biggest change in the last few years was the introduction of Bazel modules (also known as Bzlmod) and the deprecation of WORKSPACE mode. I've updated all the previous articles to be compatible with Bazel modules, but today we'll explore newly introduced functionality: how to write a module extension, and why you'd want to do so.
This "Maintaining Compatibility" trilogy began by describing how to, well,
maintain compatibility with Bzlmod builds, legacy WORKSPACE builds, and a
range of dependency versions. However, this was only half the story. Automated
testing is essential for validating our compatibility assertions, not lying to
ourselves and our users, and preventing the undoing of our hard work.
The previous post described how to write and run tests
that enable switching between Bazel versions and the Bzlmod and legacy
WORKSPACE build modes. Those tests use the latest versions of our non-Bazel
dependencies to ensure forward compatibility.
This fourth and final part of our trilogy describes how to write tests to
validate backwards compatibility with combinations of older dependency
versions. We'll build on the techniques from the previous post, while learning
what makes these backwards compatibility tests substantially different from
other tests in the suite.
We've covered techniques for ensuring that your project remains compatible with
different Bazel versions, both Bzlmod and legacy WORKSPACE builds, and older
dependency versions. However, we shouldn't make any promises until we've
validated that these properties actually hold, preferably via automated testing
and continuous integration.
This third post in our four part trilogy covers writing Bazel tests that
allow for flexibly switching between various Bazel configurations. We'll
consider advice on how to run the tests locally while developing and how to run
them in continuous integration.
In the previous post, we reviewed guidelines for
maintaining compatibility with both Bzlmod and legacy WORKSPACE builds, and
older and newer dependency versions. I promised that in this post and the next,
we'd discuss testing approaches to help ensure that this remains the case.
However, a discussion in the Bazel Slack workspace has revealed a Bzlmod and
legacy WORKSPACE compatibilty issue I'd missed in the previous post. So in
this post, I'll discuss what to do with the class of legacy WORKSPACE
configuration macros that use Label with computed repository names.
As we'll see, this one issue alone ended up meriting a substantial post in
itself. We'll cover adding dependency attributes to repository rules, generating
.bzl files to resolve Labels, chaining together module extensions, and using
macros in generated BUILD files. The former two options are relatively
straightforward, but we cover the latter two options in case your use case
requires them.
Such legacy WORKSPACE macros commonly seem to pertain to toolchain
configuration, selecting repositories to instantiate based on user defined
parameters. So we'll use a small example project to illustrate these solutions
as they apply specifically to toolchain configuration.
As we're well aware by now, Bzlmod is the new hotness, and WORKSPACE is old and
busted and going away in Bazel 9. However, Bazel 8 still supports
WORKSPACE, and thus legacy WORKSPACE usage won't completely disappear for
some years yet. Despite the Bazel community's efforts to help facilitate Bzlmod
migrations (including this blog series), some projects may remain unable to
migrate sooner than later.
What's more, publishing your repository for use by other projects raises the
challenge of supporting a range of older and newer versions of its dependencies.
Your repository should work with the newest versions of Bazel, rules_java,
protobuf, and other dependencies in order to stay current. However, not every
project that could benefit from the latest version of your repository will want
to upgrade these other dependencies right away.
This post describes how to design a repository to remain compatible with both
WORKSPACE and Bzlmod, and with newer and older dependency versions. Just as
importantly, in the next two posts, we'll discuss testing approaches to help
ensure that this remains the case.
Part of the promise of Bazel modules is that they are largely self-initializing
in an order-independent way. Rule sets, in particular, no longer need to burden
users with importing and invoking macros to instantiate repositories and
toolchains in a specific order. This burden now shifts to rule set maintainers,
but the existing implementation may not provide this ease of use without
modification.
This post describes the introduction of a new "toolchainized" API for
rules_scala v7.0.0 that better encapsulates toolchain configurations and
dependencies. We'll see how this new design enables optimal Bzlmod
compatibility, while simultaneously shrinking the legacy WORKSPACE API surface
without losing functionality. We'll also see how the Bzlmod and legacy
WORKSPACE APIs provide similar interfaces while sharing the same underlying
implementation, facilitating Bzlmod migrations.
The apparent and canonical repository name schema under Bzlmod is the gift that
keeps on giving. Much of what it has to give is quite good—once you learn
how to really hold it right. Which is to say, to avoid holding canonical repo
names at all. That's what the three previous "Repo Names..." posts in this
series were all about.
Those previous posts, however, pertained to usingBUILD rules, or when
accessing runfiles. In those situations, solutions exist to avoid handling
canonical repo names directly as a consumer.
If you maintain a Bazel rule set, or need to fix a rule set upon which your
project depends, this is the post for you. We'll see how improper repo name
usages sneak into rule implementations, and how to shoo them out. Examples
include removing canonical repo names from embedded resource paths, filtering
lists of target labels, and generating default repository target names. We also
discuss removing internal references to your project's own apparent repository
name to avoid minor yet preventable issues.
In the previous Bzlmod post, we covered writing your own
Bazel module extensions to adapt your own setup code for dependencies that
aren't Bzlmod compatible. However, there are other Bzlmod incompatibilities and
related breakages that module extensions alone can't fix, such as forbidden API
usage or Windows path length errors.
This post shows you how to patch your dependencies and covers several situations
where patching is the only solution. We'll describe how to create and apply
patches for your dependencies, if you can't wait for upstream fixes (or
contribute them yourself).
So far, we've covered relatively easy Bzlmod fixes, how to hold runfiles and
pkg_tar correctly, and how to access canonical repo names when absolutely
necessary. Now we'll discuss replacing WORKSPACE statements with your own
module extensions. You can use them to wrap your project's setup macros, and to
adapt external repositories that aren't Bzlmod compatible to work with your
Bzlmodified project.
Between roughly 2006 to 2008, Google developed remote caching and execution
technologies to scale its massive monorepo based software development
operation. This platform included Forge, the remote caching and execution
platform, and Blaze, a tool for building large multilanguage software projects,
eventually open sourced as Bazel. The advantages of this original platform
were so obvious that it literally sold itself, and ultimately inspired the
EngFlow platform.
Today, EngFlow is one of several competing remote caching and execution products
now available in the commercial space. This post describes how we continuously
benchmark our own product against different configurations to ensure that we
offer the best possible value. We hope that sharing our methodology might help
you evaluate whether remote caching or execution is right for your organization.