Skip to content

Migrating to Bazel Modules (a.k.a. Bzlmod) - Fixing and Patching Breakages

In the previous Bzlmod post, we covered writing your own Bazel module extensions to adapt your own setup code for dependencies that aren't Bzlmod compatible. However, there are other Bzlmod incompatibilities and related breakages that module extensions alone can't fix, such as forbidden API usage or Windows path length errors.

This post shows you how to patch your dependencies and covers several situations where patching is the only solution. We'll describe how to create and apply patches for your dependencies, if you can't wait for upstream fixes (or contribute them yourself).

This article is part of the series "Migrating to Bazel Modules (a.k.a. Bzlmod)":

I occasionally update these blogs based on feedback, noting the changes in the Updates section at the bottom whenever I do. So don't forget to check the earlier blog posts every so often for new and improved information!

Prerequisites

As always, please acquaint yourself with the following concepts pertaining to external repositories if you've yet to do so:

As a quick refresher, here are two ways to generate a patch (both produce the same output format):

  • git diff produces a patch with changes against the current HEAD.
  • diff -urN a b generates a patch between any two directories a and b.

Assuming a patch with the file name foo.patch, patch -Np1 -i file.patch applies the patch in the current directory.

Patches for problems module extensions can't solve

Continuing the story from the previous post, rules_scala had other Bzlmod incompatibility issues that couldn't be fixed with only a module extension. Many of these were related to repo name handling. (I've since contributed fixes for these Bzlmod breakages in #1621, #1650, and #1694, but they aren't yet available in an official release.) We also had a problem with long repository names in rules_rust on Windows, and with a pre-Bzlmodified vendored copy of protobuf.

I'll share how I solved these deeper Bzlmod incompatibilty problems by generating my own patches for these external dependencies. Hopefully these changes inspire insights that can help solve the problems you face.

How to generate and apply a patch to a Bazel module or other repo

The basic process for creating patches and applying them to dependencies is:

  1. git clone the repository you wish to modify to your local machine.
  2. Within the cloned repository, git checkout the tag corresponding to the version you wish to target with your patch.
  3. Add a MODULE.bazel file to the cloned repository if it doesn't already have one.
  4. In your project, immediately after the bazel_dep() declaring the dependency in MODULE.bazel, use local_path_override() to reference the cloned repository.
  5. Update the cloned repository until your project builds successfully.
  6. Use git diff (or whatever method you prefer) within the cloned repository to generate a patch.
  7. Save the patch file in your project's repository.
  8. Replace local_path_override() with single_version_override() to reference the actual repository distribution and apply the patch.

For example, this is how I use local_path_override() in EngFlow/example to find my local bazelbuild/rules_scala repo:

Using local_path_override() in MODULE.bazel
1
2
3
4
5
bazel_dep(name = "rules_scala")
local_path_override(
    module_name = "rules_scala",
    path = "../../bazelbuild/rules_scala-bzlmod"
)

Here is how I use single_version_override() to apply a patch to protobuf in the 'bzlmod-enable' branch of my mbland/rules_scala repo:

Using single_version_override() to apply a patch
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
bazel_dep(
    name = "protobuf",
    version = "30.1",
    repo_name = "com_google_protobuf",
)

# Temporarily required for `protoc` toolchainization until resolution of
# protocolbuffers/protobuf#19679.
single_version_override(
    module_name = "protobuf",
    patch_strip = 1,
    patches = ["//protoc:0001-protobuf-19679-rm-protoc-dep.patch"],
    version = "30.1",
)

If the dependency does not have a MODULE.bazel file, and you're not ready to add one yourself, make the following adjustments:

If you have many dependencies to patch, you can also store your patches in a custom Bazel registry and remove the local_path_override(). Bazel will automatically apply the patches from the registry. In general, however, it's better if your repo shows that you've patched the dependency, and the patch is committed to version control.

Use non-registry overrides or a custom registry for unpublished rules.

As Manuel Naranjo suggested in a #bzlmod thread in the Bazel Slack workspace, archive_override() can import Bazel modules that aren't published to a registry. Other non-registry overrides, and repository rules like local_repository() that don't access the Bazel Central Registry or other public server, can serve this purpose as well.

Patches can produce a maintenance burden

Patch files can be difficult to maintain. They can conflict with upstream changes, requiring you to checkout the new version, apply the patch, resolve conflicts, test, and generate a new patch. It's better to upstream your patched changes if possible, then drop the patch!

Avoiding native.register_toolchain() calls

Among the most common WORKSPACEisms that break Bzlmod compatibility are macros that call native.register_toolchain(). Under Bzlmod, register_toolchain can only appear in the MODULE.bazel file. Borrowing from the "Module extensions" section of the previous post:

WORKSPACE MODULE.bazel
Allows native.register_toolchains() calls, including in *_toolchains() macros. The main repository's WORKSPACE file must contain calls that register all necessary toolchains. Requires register_toolchains() in MODULE.bazel; native.register_toolchains() in an extension raises an error. Each module's MODULE.bazel file can register its own toolchains automatically. The root module need only call register_toolchains() for a dependency's toolchains to customize toolchain resolution.

To fix this, first copy the register_toolchain() statement (without native.) from the macro into your MODULE.bazel file. If the macro consists solely of native.register_toolchain(), and no module extensions call it (directly or indirectly), you're done. No patching necessary.

Otherwise, if the macro also contains other logic required by a module extension, then do one of the following:

  1. If the macro has a register_toolchains parameter or similar, call it from your own module extension with register_toolchains = False in the argument list. No patching necessary.
  2. If the macro unconditionally calls native.register_toolchains(), either remove the code or add a register_toolchains parameter and call it with register_toolchains = False. Create a patch for the updated macro definition.

Avoiding native.bind() calls

native.bind() is completely disallowed under Bzlmod.

WORKSPACE MODULE.bazel
Allows bind() and native.bind() calls. bind() and native.bind() raise an error. Requires removing bind() targets and updating dependents to depend upon apparent repository name labels or alias() targets instead.

If your module extension calls macros containing native.bind(), here are your options, all of which require patching the dependency:

  1. Remove the native.bind() call altogether.
  2. Replace the original native.bind() call with an alias, possibly using native.alias(), and using the same name and actual parameters.
  3. If a macro has a bind_targets parameter or similar, call the macro from your module extension with bind_targets = False. Depending on the original bind() target, you may need to create a new alias target.
  4. If you're afraid to remove the native.bind() call completely, add a bind_targets = True parameter or some such and call it with bind_targets = False. You may still need to add an alias target.

In each of these cases, you may need to update any dependencies on the original //external: target with the actual target or an alias. If the //external: target appears in your dependencies, you will need to patch those instances as well.

Adding a MODULE.bazel file to a vendored repo to resolve conflicts

At one point, we had a vendored copy of the protocolbuffers/protobuf library, a ubiquitous dependency in the Bazel ecosystem, that wasn't Bzlmod compatible. At the same time, the bazel_tools module baked into Bazel depends on the protobuf Bazel module. This caused our Bzlmod builds to fail due to a combination of factors.

Since performing the below steps during our initial Bzlmod migration, we've since upgraded to a properly Bzlmod compatible version of protobuf. However, if you're in a similar situation with another dependency, hopefully this information provides helpful insights.

What follows isn't exactly a patch based solution, but it's in the ballpark, since applying a change to vendored repos is akin to patching.

Try override_repo from Bazel 7.4.0 when using module extensions.

override_repo() (in Bazel 7.4.0 and later) enables the root module (i.e., the main repository's MODULE.bazel file) to override repos defined by a module extension. In this case, however, there weren't module extensions involved, and the problem ultimately called for overriding the entire module.

The first part of the problem was that we set the following in our main repo's .bazelrc so that unused C++ functions break the build:

Setting -Werror=unused-function in our .bazelrc
1
2
build:clang --cxxopt="-Werror=unused-function"
build:clang --host_cxxopt="-Werror=unused-function"

In our vendored protobuf, we overrode this in its //build_defs:cpp_opts.bzl:

Disabling unused function errors in vendored protobuf
1
2
3
4
5
6
7
8
9
COPTS = select({
    # ...snipped other conditions...
    "//conditions:default": [
        "-DHAVE_ZLIB",
        "-Woverloaded-virtual",
        "-Wno-sign-compare",
        "-Wno-unused-function",
    ],
})

When beginning our Bzlmod migration, Bazel attempted to build two distinct protobuf versions configured for our project:

Bazel attempted to build both protobuf versions, since one was a WORKSPACE repo and one was a module, so neither instance overrode the other. Both versions contained the unused Offset() function. The build then failed with an unused function error when Bazel would try to build the protobuf Bazel module imported by bazel_tools:

Our protobuf originally broke under Bzlmod.
1
2
3
4
5
external/protobuf~/src/google/protobuf/generated_message_tctable_lite.cc:347:14:
  error: unused function 'Offset' [-Werror,-Wunused-function]
inline void* Offset(void* base, uint32_t offset) {
             ^
1 error generated.

protobuf 21.7 was the highest Bazel module version available at the time. I tried importing it with bazel_dep() and patching it with single_version_override() to add -Wno-unused-function to COPTS in build_defs/cpp_opts.bzl. This did not override the protobuf module repo from @bazel_tools with our copy, so Bazel built both protobuf modules, causing a duplicate Java class error. (This may have been related to the problem reported in bazelbuild/bazel: duplicate repos between WORKSPACE and MODULE when using repo_name mapping #21818.)

The solution that worked at the time was adding our own custom MODULE.bazel file to our vendored protobuf copy and using local_path_override():

Using our protobuf clone in MODULE.bazel
1
2
3
4
5
bazel_dep(name = "protobuf", repo_name = "com_google_protobuf")
local_path_override(
    module_name = "protobuf",
    path = "third_party/protobuf",
)

With our vendored copy acting as a proper Bazel module, Bzlmod ensured it was used throughout the dependency graph, including in the bazel_tools module. I won't go into the details of that MODULE.bazel file, since newer protobuf versions now contain their own.

Sharing vendored repositories with a vendored module

This solution also required adding a module extension to our project to share other vendored repositories between protobuf and the root module. It instantiates the other vendored repos using local_repository().

Module extension for instantiating and sharing vendored repos
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
load("@bazel_tools//tools/build_defs/repo:local.bzl", "local_repository")

_REPOSITORIES = [
    "foo",
    "bar",
    "baz",
]

def _protobuf_deps_impl(_ctx):
    for repo in _REPOSITORIES:
        local_repository(
            name = repo,
            path = "third_party/{}".format(repo),
        )

protobuf_deps = module_extension(
    implementation = _protobuf_deps_impl,
)

The root module and protobuf then bring these repos into scope via use_extension() and use_repo().

Using the module extension to bring shared repos into scope
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
# Enables the vendored repo to access the extension. "main-repo" is a fake name
# representing whatever the `module()` name of the main repo happens to be.
bazel_dep(name = "main-repo", version = "0.0.0")

shared_deps = use_extension("@main-repo//:shared-deps.bzl", "shared_deps")

use_repo(
    shared_deps,
    "foo",
    "bar",
    "baz",
)

override_repo() and inject_repo()

override_repo() or inject_repo() enable the root module to choose its own repositories for use by a dependency's module extension:

  • override_repo() is appropriate for overriding a repository of the same name as one already used by a dependency, and doesn't require patching.
  • inject_repo() is appropriate when introducing a new repo while patching a dependency.

Prefer one of these to writing a custom module extension when a vendored repo already has a module extension for instantiating repos or applying patches.

Hacking around Windows path length issues due to longer canonical repo names

After enabling Bzlmod, our Windows continuous integration build failed due to a problem when building the rules_rust Bazel module:

rules_rust breakage due to long paths on Windows
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
error: linking with `C:/Program Files (x86)/Microsoft Visual
Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/HostX64/x64/link.exe`
failed: exit code: 1181
|
= note: "C:/Program Files (x86)/Microsoft Visual
Studio/2019/BuildTools/VC/Tools/MSVC/14.29.30133/bin/HostX64/x64/link.exe"
[ ...snip... ]
= note: LINK : fatal error LNK1181: cannot open input file
'bazel-out\x64_windows-opt-exec-ST-899f5471b48c\bin\external\
  rules_rust~~rust~rust_windows_x86_64__x86_64-pc-windows-msvc__stable_tools\
  rust_toolchain\lib\rustlib\x86_64-pc-windows-msvc\lib\
  librustc_std_workspace_alloc-3928c78544f9c50c.rlib'

It turns out that path is 239 characters long. It comes under the apparent 260 character limit, but appears to grow past the limit after it's converted to an absolute path internally.

There at least two existing issues reflecting this problem:

Apparently another fix exists, if we knew where to patch it in:

The maintainers originally made the decision to disable the rules_rust Windows builds instead, due to a lack of a dedicated Windows maintainer. This remained true as of rules_rust version 0.54.1, though bazelbuild/rules_rust#3023 restored the builds for rules_rust 0.55.0. (There's still an open issue to renable Windows crate_universe example tests at some point.)

At the time, we used rules_rust 0.41.1, and I realized that these platform specific path segments contained largely redundant information. I created the following patch to remove the middle component, known as the target triple (or triplet). (I've stripped out the comments I'd included in the original, which I used almost verbatim to write this section of the post.)

Diff
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
diff --git a/cargo/private/cargo_bootstrap.bzl b/cargo/private/cargo_bootstrap.bzl
index beaac8ac..55f5447c 100644
--- a/cargo/private/cargo_bootstrap.bzl
+++ b/cargo/private/cargo_bootstrap.bzl
@@ -268,7 +268,7 @@ cargo_bootstrap_repository = repository_rule(
                 "`{system}` (eg. 'darwin'), `{channel}` (eg. 'stable'), and `{tool}` (eg. 'rustc.exe') will be " +
                 "replaced in the string if present."
             ),
-            default = "@rust_{system}_{arch}__{triple}__{channel}_tools//:bin/{tool}",
+            default = "@rust_{system}_{arch}__{channel}_tools//:bin/{tool}",
         ),
         "rust_toolchain_rustc_template": attr.string(
             doc = (
@@ -277,7 +277,7 @@ cargo_bootstrap_repository = repository_rule(
                 "`{system}` (eg. 'darwin'), `{channel}` (eg. 'stable'), and `{tool}` (eg. 'rustc.exe') will be " +
                 "replaced in the string if present."
             ),
-            default = "@rust_{system}_{arch}__{triple}__{channel}_tools//:bin/{tool}",
+            default = "@rust_{system}_{arch}__{channel}_tools//:bin/{tool}",
         ),
         "srcs": attr.label_list(
             doc = "Souce files of the crate to build. Passing source files here can be used to trigger rebuilds when changes are made",
diff --git a/crate_universe/private/crates_repository.bzl b/crate_universe/private/crates_repository.bzl
index 0a3de518..60bfd06b 100644
--- a/crate_universe/private/crates_repository.bzl
+++ b/crate_universe/private/crates_repository.bzl
@@ -288,7 +288,7 @@ CARGO_BAZEL_REPIN=1 CARGO_BAZEL_REPIN_ONLY=crate_index bazel sync --only=crate_i
                 "`{system}` (eg. 'darwin'), `{cfg}` (eg. 'exec'), `{channel}` (eg. 'stable'), and `{tool}` (eg. " +
                 "'rustc.exe') will be replaced in the string if present."
             ),
-            default = "@rust_{system}_{arch}__{triple}__{channel}_tools//:bin/{tool}",
+            default = "@rust_{system}_{arch}__{channel}_tools//:bin/{tool}",
         ),
         "rust_toolchain_rustc_template": attr.string(
             doc = (
@@ -297,7 +297,7 @@ CARGO_BAZEL_REPIN=1 CARGO_BAZEL_REPIN_ONLY=crate_index bazel sync --only=crate_i
                 "`{system}` (eg. 'darwin'), `{cfg}` (eg. 'exec'), `{channel}` (eg. 'stable'), and `{tool}` (eg. " +
                 "'cargo.exe') will be replaced in the string if present."
             ),
-            default = "@rust_{system}_{arch}__{triple}__{channel}_tools//:bin/{tool}",
+            default = "@rust_{system}_{arch}__{channel}_tools//:bin/{tool}",
         ),
         "rust_version": attr.string(
             doc = "The version of Rust the currently registered toolchain is using. Eg. `1.56.0`, or `nightly/2021-09-08`",
diff --git a/rust/repositories.bzl b/rust/repositories.bzl
index 321f8448..78390fd6 100644
--- a/rust/repositories.bzl
+++ b/rust/repositories.bzl
@@ -986,7 +986,7 @@ def _get_toolchain_repositories(name, exec_triple, extra_target_triples, version
         # Define toolchains for each requested version
         for channel in channels.values():
             toolchain_repos.append(struct(
-                name = "{}__{}__{}".format(name, target_triple, channel.name),
+                name = "{}__{}".format(name, channel.name),
                 target_triple = target_triple,
                 channel = channel,
                 target_constraints = target_constraints,

The savings were substantial, and fixed our Windows builds:

Configuration Path Component Length
Before, under bzlmod rules_rust~~rust~rust_windows_x86_64__x86_64-pc-windows-msvc__stable_tools 74
Before, under WORKSPACE rust_windows_x86_64__x86_64-pc-windows-msvc__stable_tools 57
After, under bzlmod rules_rust~~rust~rust_windows_x86_64__stable_tools 50

The "proper" fix may still be to find places to inject the UNC path prefix, or coaching users to enable long paths on Windows somehow. But in the meanwhile, this got our own builds working again quite quickly and reliably.

A confession...

I've been meaning to try contributing this change upstream, to see if the maintainers would accept it. I may yet do that, though bazelbuild/rules_rust#3023 seems to work for now.

protobuf is dropping Windows + MSVC support

On a related note, protobuf is dropping support for Windows Microsoft Visual C++ builds because of the path length issue. As mentioned in the Conclusion, I'll write more about this eventually.

Conclusion

Writing my own MODULE.bazel files, module extensions, and patches for external dependencies helped me complete EngFlow's Bzlmod migration. I'm hopeful that sharing these insights and techniques will help others make progress on their own migrations without waiting for all their dependencies to migrate.

I've mostly prioritized getting our Bzlmod migration done and publishing about the process first, but I'm also now trying to contribute changes upstream. I've almost finished contributing to the Bzlmodification of rules_scala, including the changes described in these blog posts and then some. Contributing upstream has taken a lot more time, but helps a lot more people, which I find rewarding in ways that transcend working only internally. Even so, these contributions wouldn't've been possible without having gone through this process, as I learned a lot and had working production code in hand.

For the next post, I'll cover more examples of repo name dependencies and how to resolve them. Technically, these are also patchable problems, but there's enough of them that they warrant yet another dedicated post.

I'm no longer sure I'll write the rules_nodejs post mentioned in previous posts. While I dove deeply into Bzlmodifying rules_scala, my colleague Ricard Solé migrated our code from my heavily patched 5.8.5 to aspect_rules_js. It's not officially canceled, but its fate is uncertain.

However, here are additional Bzlmod related topics that I'll very likely cover:

As always, I'm open to questions, suggestions, corrections, and updates relating to this series of Bzlmodification posts. It's easiest to find me lurking in the #bzlmod channel of the Bazel Slack workspace. I'd love to hear how your own Bzlmod migration is going—especially if these blog posts have helped!