Benchmarking any2bazel - towards more efficient migrations

We’re fond of Bazel, and particularly fond of the capabilities it offers developers. Faster builds, made more efficient through remote execution and caching, made more understandable through declarative configuration, queryability, and precise logging – these deliver benefits and solutions to developers once Bazel’s all set up and integrated into their daily processes.

But it’s a rare developer who enjoys spending their time and effort in build system configuration. Wouldn’t we rather be solving our customers’ problems? And the decision to migrate from an existing build system to Bazel is one that demands consideration of huge trade-offs; the labor and opportunity costs of engineering time spent on migration and adapting internal workflows to a new build system discourage the pursuit of its advantages.

In early 2026 I became interested in AI-guided Bazel migration because the capabilities of models like Opus 4.5 and agentic tools like Claude Code seemed to offer a way of realizing those benefits without so much toil – that is, if AI-guided Bazel migration worked. I believed it would work eventually, if not at that point in time, given the rapid pace of capability growth in AI models and their harnesses. I also believed it would be possible to distill lessons learned from open-source projects’ Bazel migrations and AI-guided migration attempts into a skill. If done right, such a skill could elevate those capabilities, and the process of improving the skill with better distillation from more and more migration attempts could augment the "natural" capability growth coming from the models getting smarter.

I was particularly struck by the question, "how would we know when we arrived?" It seemed answerable, if only we had an experimental laboratory that evaluated AI-guided Bazel migration attempts.

Many months later, around the time any2bazel launched, I began work on any2bazel-lab, to bring experimental results to bear on the question of how well current AI models performed at automated Bazel migration, both with and without the any2bazel skill. Anthropic’s frontier models – a mix of Opus 5, 5.5, Fable 5.1, and a bit of Haiku 4.5 – extensively contributed to the lab’s development, and coordinated the execution of its migration experiments. Our experiments are ongoing. Here’s a summary of what we’re learning, and how.

How any2bazel-lab works

In our latest and largest study, the lab attempts two automated migrations of a given project. Both runs are performed by the same AI model and agent harness, using initial prompts that are nearly identical, in identical environments. The differences are only that one run’s initial prompt invokes the skill using /any2bazel, and in the other the lab removes the skill from disk beforehand because it triggers on the prompt text even absent its literal invocation. These are "one-shot" runs: the agent attempting a migration has no subsequent interactions with a human or outside agent past its initial prompt. We call the run using the skill the "skill" run, and the one without the skill the "no-skill" run.

Once each run finishes, the lab measures it with tools that compare its actions and final states against a reference build of that project, performed using the project’s preexisting build system. The lab also estimates the token cost and measures the duration of each run.

Although I wrote "the lab" a few times there, if I wanted to be more precise I would write "a frontier AI model such as Opus 5.5, running in a coding agent such as Claude Code, following precise instructions in the lab’s documentation, including running programs developed to help administer experiments." This "outer" AI agent never itself performs migrations. It coordinates the "inner" AI agents that do.

Those precise instructions include preregistered hypotheses and decision rules, established before each study’s runs. Not that these were perfect – but every amendment we made thereafter was documented, with an explanation of why we adopted it and our understanding of what it affected. Also, whenever we discovered an issue with the lab’s tools or analysis that meant some record ought to be corrected, that correction was itself recorded in an errata list.

So far we’ve attempted migrations of 14 projects of varying size and complexity using the lab. All use CMake as their original build system. The migrations all used Anthropic’s Claude Opus 5 for their AI model, along with Claude Code pinned at version 2.1.270. In a prior study we repeated the migration of a few smaller projects to learn about how they vary due to AI model nondeterminism. Our large project repetitions have been limited but we intend to do more. We also intend to look at projects using other build systems; next on our docket are Gradle projects.

However, we’ve already learned some interesting things about what current models are capable of, and how the any2bazel skill affects their performance.

Migrations had fewer error causes and narrower cost ranges with the skill, on smaller projects

In our first study, we ran migrations on four smaller projects both with and without the any2bazel skill, and repeated each run three times. Across the 12 runs using the skill, only one included an error that affected its build’s output’s behavior. Without the skill, these types of errors happened in each run of the lensfun project and two out of three runs of the pono project.

The range of costs for skill runs were narrower and tended to be cheaper than those for the no-skill runs.

Each dot is one S01 run; the line spans an arm's cheapest to most expensive run. Log scale.

Migrations with the skill finished faster and more cheaply

For example, bitcoin finished at 24.1 minutes and $9.00 with the skill vs. 32.0 minutes and $13.87 without; openttd finished at 17.1 minutes and $8.70 with the skill vs. 48.3 minutes and $26.24 without. On projects above their size, skill runs were still usually cheaper, but often because they did less, as the next sections show. (Project migration case studies, and additional analysis, can be found in our detailed progress report here.)

Each row is a project; each dot is a skill or no-skill run. Coverage and tests ported are shares of the reference; behavioral causes are distinct decisions that change what the program does; cost is notional and time is the migration wall time. A hollow dot is a run without a whole build: judged on what builds; × marks a run with no build to judge. The tests panel is blank where a run migrated no tests: a partial build, or a project without tests.

Migrations with the skill tended to make fewer build decisions that changed how the output behaves

Across the thirteen projects we could compare, the run with the skill made fewer such choices on seven projects, tied on five projects, and made more on one: mysql-server, where the skill run’s build couldn't stand on its own.

Above a size threshold, the skill’s tools contributed but weren’t sufficient

The any2bazel tools give a migrating agent a measure of convergence. It wasn’t enough to ensure that bazel build //... would succeed; for 3 of the 10 larger projects, the skill run produced a build that failed that check while the no-skill run passed it. The skill runs came close in two of them: gdal built 280/283 targets, xbmc built 261/269 (but its main program was one of the failing targets). The mysql-server skill run "migrated" its CMake build to Bazel by having its Bazel build depend on the CMake build’s outputs.

In a fourth of the 10 larger projects, yugabyte-db, the skill run also ended without a Bazel build, but in a noteworthy way...

The skill provided at least a little wisdom (in one run) about how much is too much to attempt

Our experiment with the yugabyte-db project had an unexpected result. The skill run analyzed yugabyte-db’s CMake build and its agent observed several blockers – the strongest of which was that the original build depended on a non-CMake build of a PostgreSQL fork, which the agent concluded it was insufficiently equipped to handle. That run declined the migration, and made its decision in less than 3 minutes and at an estimated cost of $1.19. Its no-skill counterpart charged boldly ahead, worked for 95 minutes at an estimated cost of $42.23, and produced a migration that compiled most of the project itself but linked the PostgreSQL backend from the CMake build. Its bazel build //... succeeded, but its servers aborted on startup, and 393 of the 573 tests that passed in the original build failed.

This was an admirable outcome for the skill, compared to the migration that ran without it. But, a result being hard to criticize does not mean it’s successful. This experience, along with several of the other pairs that ran on large projects, suggests that the skill has room to grow.

Recursive (self?) improvement

Lots of good ideas emerged naturally from our benchmarking. We intend to upgrade any2bazel with at least some of them, and the benchmarking lab provides us with a framework for measuring whether the changes are helpful. As we add more projects to our set of migration candidates, and run more migration attempts across them, we hope to make any2bazel more valuable to its users, until the day when AI models and agents Solve Programming Forever and Ever.

To start with:

• The skill should ask the migrating agent to port tests.
• A finished migration should pass the bazel build //... check!
• A migrating agent should target, at least, the configuration that the preexisting build documents as its default. If the Bazel build it produces requires a particular configuration command, that command should be recorded in a structured way.
• The skill should guide fully autonomous agents towards healthy ways of resolving migration issues when possible, where the skill currently instructs agents to stop and consult their user. The latter may make more sense in interactive use cases. In fully autonomous runs, like the ones conducted by any2bazel-lab, the agent almost always barrels straight through the skill’s stopping conditions and sometimes makes questionable decisions on the other side (e.g. configuring Bazel builds to depend on CMake outputs!)

... among others. The lab’s work produced 25 specific proposals so far, and I anticipate many more, as candidate project diversity widens.

What this doesn’t show

What we’ve learned so far from our any2bazel-lab experiments is more anecdotal than statistical. Several caveats should qualify our observations:

• We’ve only attempted migrations once per skill/no-skill arm for almost all of the medium sized and up projects.
• We ran two large and extra large projects twice, cmake and mysql-server. On mysql-server, we measured greater differences between runs within each arm than between arms. On cmake, the skill arm’s two runs were close on coverage, causes, and cost, but not on its configure headers. This strongly suggests that an anecdotal and qualitative perspective towards these runs is better justified than any statistical perspective.
• We’ve only tested with Anthropic’s Claude Opus 5. Opus 5.5 released in the middle of our latest batch of runs, and we continued using Opus 5 for our migrating agents deliberately, for consistency. We have not experimented with using Opus 5.5, or any other AI models, for Bazel migrations in any2bazel-lab.
• The version of any2bazel we tested, commit 210b25a made on 2026-09-11, should be considered a very early version of the skill. It has a long road ahead.
• We’ve only tested the skill in an unattended fashion. Interactively using the skill has helped with larger migrations (as noted here), but this is harder to systematize and automate for benchmarking purposes.
• The version of any2bazel-lab we used should also be considered very early, and evolved significantly throughout our experimentation. Our hope is to open-source this soon, so that our work can be checked (and maybe contributed to!)
• Token costs are estimates, not actual measured costs. Almost all of this work was conducted with quota from a subscription.

More information about the migrations we ran in these studies can be read in our detailed report, including metrics for each run and descriptions of what the automated migrations got wrong.

What's next

Additional experiments! We intend to benchmark migrations to Bazel from build systems besides CMake, the effects of our ongoing development on the any2bazel skill, the effects of equipping the migrating AI agents with the ability to run builds with RBE (Remote Build Execution), and AI models beyond Opus 5. Look forward to more blog posts covering those results, and a link to any2bazel-lab itself as soon as it's open-sourced on GitHub.