Benchmarking any2bazel

A report from the any2bazel lab · migrations of CMake projects to Bazel by Claude Opus 5, with the any2bazel skill and without it · written 2026-09-26 by the lab's supervising agent (Claude Opus 5.5) for Mark Schaller · every number on this page is rendered from the lab's run records by reports/v2/facts.py

1. Summary

The lab gives the same instruction ("port this CMake project to Bazel") to the same model in the same container twice: once with the any2bazel skill installed, once with it removed. It then measures both results against the project's own CMake build. It did this on 14 open-source projects, from graphics and numerical libraries of a few dozen sources to Blender and MySQL at several thousand.

Where the skill arm produced a build, it usually left fewer counted behavioral causes: decisions classed as changing what the built program does. This is H1, the study's main hypothesis, and it is supported. Of the 13 pairs it counts (yugabyte-db's is left out, because its skill arm declined; below), the skill arm has fewer behavioral causes on 7, more on 1, and the same number on 5. The one "more" is mysql-server, where the skill arm has no build to judge and the rule counts that as worse.

But the skill arm failed to produce a build more often. The no-skill arm's bazel build //... succeeded on all 14 projects. The skill arm's succeeded on 10. Of the rest:

• 2 built all but a few targets (gdal, xbmc; they are judged on the part that builds, marked †);
• 1 could not be rebuilt from its own repository (mysql-server);
• 1 was declined, correctly, with a report of why (yugabyte-db).

So M0, "the skill arm makes an additive migration aiming at parity", is inconclusive. A migration counts as additive only if it adds a Bazel build whose bazel build //... succeeds and leaves CMake working: 10 of 14. In xbmc and mysql-server, the skill arm's failing targets include the project's main program (kodi, mysqld).

The skill arm was cheaper and faster, and often did less. It cost less on 13 of 14 pairs and finished sooner on 13.

• Coverage. On the 12 pairs where both arms have a coverage figure, the no-skill arm covered more of the reference's sources, by more than 1% of them, on 3 (xbmc, qgis, freecad). The skill arm did so on 0. On the other 9 the two are within 1%.
• Tests. On the 9 pairs where both arms were tested, the no-skill arm ported more of the reference's test cases, by the same margin, on 2 (qgis, freecad), and the skill arm on 0. On the 4 pairs where the skill arm had no whole build to test (mysql-server, gdal, xbmc, yugabyte-db), the no-skill arm ported tests on 4.
• Passing. Where ported tests could be matched, they passed, with one exception (yugabyte-db's no-skill arm); H3 is supported.

These are single runs. Each project was migrated once per arm. The two projects run twice show how much a second run can differ (§7). On mysql-server, the spread between two runs of the same arm was larger than the difference between the arms. Read every per-project statement as "in this run".

The 14 pairs

How to read the table.

• Tier: project size, from S (small) to XL (§2).
• Reference sources: the distinct source files the project's own CMake build compiles.
• Coverage: the share of those the migration also compiles.
• Behavioral causes: the number of distinct decisions that change what the built program does (§4).
• Cost: list-price notional dollars for the model tokens the run used.
• Time: launch to exit.
• partial †: the whole build fails, and the run is judged on the part that builds.
• failed: the lab could not rebuild the migration from its own repository, so no coverage, cause or test figure is judged (§6.7 gives how much of it builds); for H1 it counts as worse than any number.
• declined: the agent stopped with a report instead of a migration.

2. What was tested, and by whom

The tool. any2bazel is a skill for Claude Code: instructions and scripts that tell a coding agent how to port a CMake project to Bazel and how to check the port. Its central idea is parity. The skill extracts two models, one of CMake's build (from CMake's File API) and one of Bazel's (from bazel aquery), and a differ compares them compile command by compile command. The agent iterates until the differ reports nothing, which the skill calls converged. Every run in this report, S01's and the pilot's included, used the skill at commit 210b25a (2026-09-11), Claude Code 2.1.270, and Claude Opus 5 as the migrating model.

The lab. It is a separate repository that runs migrations and measures them. A supervising agent chooses a project and builds the project's own CMake build as the reference. It then launches an isolated migrating agent, does not help it, and judges what the agent left. Mark Schaller, who consults for EngFlow, directs the lab; the supervising agent was Claude Opus 5 until 2026-09-22 and Claude Opus 5.5 since.

What "without the skill" means. The skill's description triggers on a request to port a CMake project, so leaving its name out of the prompt does not keep it out. In the no-skill arm, the skill is moved off the disk before the run, and a check after every run confirms the agent never saw it. So the two arms of a pair run one after the other, not at the same time.

Unattended. Every run is non-interactive: nobody answers the agent. The skill's first step asks the user to confirm the plan, and in this mode the question goes unanswered and the agent carries on (F2). Its later instruction to stop at constructs it does not support is likewise only as strong as the agent's reading of it.

Terms. The rest of the report uses these terms.

• Arm: the runs with the skill (skill arm) or without it (no-skill arm).
• Pair: one run of each arm on the same project, at the same commit, in the same container image.
• Reference: the project built by its own CMake, configured the way the lab specified before either arm ran.
• Source coverage: the share of the reference's distinct compiled sources that the migration also compiles.
• Cause: one decision behind a set of differences between the two builds, such as "dropped the -fvisibility=hidden flag" (§4).
• Tier: project size by the reference's translation units: S below 150, M below 800, L below 3,000, XL above. Tables give the reference's distinct compiled sources, a close but not identical count.

3. The design

3.1 The question

Does the skill change what an agent produces when asked to port a CMake project to Bazel? The counterfactual is the same agent, on the same project, without the skill. So every project is run in pairs: the same prompt (the no-skill prompt lacks only the line invoking the skill), the same model, the same commit, the same container image, and the same resource caps. The arms differ in two things: one prompt line, and whether the skill is on disk.

3.2 What is held constant

• Prompt: byte-identical across runs within an arm, checked by the tool that creates each run.
• Model and tools: Claude Opus 5, pinned; Claude Code 2.1.270; the skill at 210b25a; Bazel modules from a registry snapshot at least a week old.
• Environment: one container image per project. The agent may install packages inside it, and what it installs is recorded.
• Caps: per run, 80 GiB of memory, a CPU quota of 28 cores, and a 24-hour wall clock.
• No intervention. Once a run starts, the supervising agent does not help it, answer it, or stop it, except to protect the host.

3.3 One reference, configured before either arm runs

Each project's reference is its own CMake build, configured the way the lab's target specification says. The specification is written and the reference built before either arm runs, and both arms are judged against the same one. An agent may build a CMake comparison of its own, and many did, with different options. That changes what the agent compares itself against, not what the lab compares it against. Whether the agent kept the reference's options is recorded (the pair tables' "kept the reference's CMake options" row; H4).

3.4 One measure for each question

Two further checks cover what parity does not compare. The lab byte-compares the files the build generates. It also records the optimization level, which the skill's differ discards.

3.5 The hypotheses, and the rules that decide them

S02 was pre-registered: its targets, measures, hypotheses and decision rules were written and frozen before its first run. Its rules were later amended 16 times, each amendment dated and recorded (studies/S02-large/PREREGISTRATION.md). The rules below are the ones in force after all of them. Two, A15 and A16, were adopted after the analysis, one with its effect already known. Each corrected a measure that described migrations wrongly, and appendices B and C say what each moved.

• M0. The skill arm's migration is additive (it adds a Bazel build whose bazel build //... succeeds, and leaves CMake working) in at least 80% of its runs; refuted at 50% or less.
• H1. Counting distinct behavioral causes per run, and provided at least half the pairs are not tied (else not decidable), the skill arm has fewer in at least 70% of the pairs that are not tied, and more in at most 15%; refuted if it has more in at least 70%, and inconclusive otherwise. A skill run with no build to judge counts as having more. yugabyte-db's pair is left out, because its skill arm declined; gdal's and xbmc's skill arms are judged on the part that builds (A12, A13).
• H2. On L and XL projects whose own test driver is not ctest, the no-skill arm wires up that driver in at least 40% of runs.
• H3. A run is checkable when at least 50% of the reference's passing test cases have a counterpart, and the reference passes at least 10 cases. A checkable run is true when at least 99% of those pass. H3 is supported when at least 80% of each arm's checkable runs are true.
• H4. The share of runs whose agent changed the reference's CMake options in its own CMake comparison is higher at XL than at M and L, by at least 0.3, in both arms; refuted if it is not higher in either arm; inconclusive otherwise.
• H5. Cost rises with project size: Spearman's ρ above 0.5 in both arms.

With one run per arm per project, these are statements about patterns in this set of projects, not statistical tests.

4. How a migration is measured

4.1 Measured, not read

Every migrating agent ends with a report of what it did, and the reports are often accurate. But an accurate report can still describe a different job from the one the lab asked for. An agent that turned many of the project's features off in its own CMake comparison can truthfully report that it converged, with nothing left to fix, against that comparison (qgis's skill arm, §6). So the lab measures each migration against the reference itself. Its own copy of the skill's extractors and differ is pinned at the version under test and runs as a separate user. It does not use the agent's comparison, its settings, or its account of the result.

The measure is the skill's own instrument. That is deliberate: parity is what the skill promises, so it is what the lab checks. But it means the skill arm's agent iterated against this differ until it reported nothing, and the no-skill arm's agent never saw it. Three things limit what that asymmetry can do:

• The lab re-runs the differ against its own reference, not the agent's comparison build, so converging against a narrowed comparison earns nothing.
• It judges at Bazel's default invocation, without the agent's suppressions: the skill's configuration lets an agent declare flags, defines or targets for the differ to ignore, and the lab drops those.
• It counts causes rather than errors, so a migration cannot improve its standing by reducing how many units one decision touches.

What the asymmetry can still do is favor the skill arm on exactly the differences the differ is good at seeing, and leave both arms unexamined where it is blind (§4.3). Source coverage, the tests, the generated-file comparison and the optimization-level record do not come from the differ, and on those the no-skill arm often did better (§6).

4.2 From differences to causes

The differ compares the two builds command by command: for each source the reference compiles, whether the migration compiles it, and with the same defines, include roots and flags; for each target, whether it exists and links the same way. The comparison is not symmetric. What the reference has and the migration lacks is an error. A flag only the migration adds is usually tolerated, and a define only the migration adds is a warning (§4.3).

Error counts are not used in this report. One decision can raise one error or thousands, depending only on how many units it touches. Dropping one define that every unit carries is one decision. The count also depends on how the two builds' paths are mapped onto each other: which directory, target or generated header on one side stands for which on the other. Under this report's strategy both maps may say only that, but their entries still come from different places. The skill arm's are its agent's own, and the no-skill arm's were written by the lab before either ran. So a directory one agent invented can be mapped in one arm and not in the other (F47).

So the lab groups the errors into causes. A fixed rule table names each cause from what its errors have in common, for example build-type, feature-defines or include-roots. A cause is a kind of decision: usually one decision, sometimes several of the same kind (cmake's no-skill include-roots gathers two declarations, F47). The report counts distinct causes.

Each cause is classed by its effect on the built program:

A few causes can only be classed by looking at the artifacts, so a check runs for each:

• whether the executables are still position-independent where the reference's are;
• whether a macro is read by any source;
• whether the project's own -std= wins on the command line;
• whether generated configuration headers agree.

A check that cannot run leaves its cause behavioral, and it is counted. Absent sources are structural rather than behavioral when every one of them belongs to a dependency the reference compiles and the migration takes, at the same version, as an external module instead (A1; pono, §6.3). A source the differ can match to the reference only by its file name, because the paths differ, may be the same source moved or a different one, so it is not classed and not counted. The class describes what the difference does, not whether the agent could have avoided it.

The behavioral causes the tables name, with the rule table's own description of each. The differ discards -O and -g before comparing (§4.3), so where a description mentions optimization, the cause can record only what survives that: defines such as NDEBUG or QT_NO_DEBUG, and hardening, visibility, link-time or architecture flags.

Both arms are judged the same way, on the comparable reading:

• The only mapping allowed between the builds is path identity: which directory or target on one side is which on the other.
• Bazel is queried at its default invocation, even if the agent declared build flags of its own.
• Nothing the agent chose to suppress is suppressed.

4.3 What parity does not compare, and how the lab covers it

• The optimization level. The skill's differ discards -O and -g flags before comparing (P16), so the lab records each side's dominant level separately (the pair tables' "optimization level" row).
• A define only the migration adds. The differ records it as a warning, not an error, and causes are built from errors, unless another define error on the same unit carries it along. For NDEBUG, which switches assertions off, amendment A16 reads the builds directly and adds the cause (appendix B). Other migration-only defines that appear only as warnings remain unclassified.
• What generated files contain. Parity compares compile commands, never the files a build generates. The lab byte-compares every generated file it can pair with the reference's, after normalizing paths (the "generated files identical" row).
• Whether the program runs. Link-interface flags and runtime loading are outside the parity model (P25). Where a migration's tests match the reference's, the tests are the runtime check (§4.4).

4.4 Tests, counted by case

The lab runs the reference's test suite and records which tests pass. Then it asks which of those have a counterpart in the migration. The unit is the test case, from each binary's own list of the cases it registers; a CMake test that runs a script, or a binary that lists no cases, counts as one case. On freecad, CMake registers every gtest case separately; on bitcoin, every Boost.Test suite; on mysql-server, one merged binary of many test files. Counting CMake's registrations instead of cases made ported tests invisible, and amendment A15 corrected that (F57). A case counts as ported in either of two ways:

• its CMake test is matched by name or executable to a Bazel test target;
• a Bazel test binary registers a case of the same name.

It passes if its Bazel target passes. If the target failed, the case's own line in the output decides, for GoogleTest; other frameworks keep the target's result.

4.5 Cost and time

Cost is notional. It prices the tokens a run used at list price: input at $5.00 and output at $25.00 per million tokens, with cache writes at 1.25× (five-minute) or 2.0× (one-hour) the input price, and cache reads at 0.1×. Messages are de-duplicated by id before summing. Time is wall-clock from launch to exit. Both measure the whole run: an agent that did more of the project spent more on it. So a cost comparison between the arms of a pair compares two different jobs, unless their coverage says they covered the same reference sources; even then one arm may have done work coverage does not count, such as tests or tools the reference does not build.

5. The verdicts

5.1 M0: the skill arm does not reliably produce a migration

10 of 14 skill runs are additive, 71%, between the refutation line (50%) and the support line (80%). The 4 that are not are the runs without a whole build (§1): gdal, xbmc, mysql-server and yugabyte-db. The no-skill arm built all 14. M0 asks about the skill arm alone, so that contrast is not part of its rule, but it is the study's most unfavorable result for the skill (F49).

5.2 H1: fewer behavioral causes where the skill arm built

supported. Of the 8 pairs that are not tied, the skill arm has fewer behavioral causes in 7 (87.5%; the rule needs 70%) and more in 1 (12.5%; the rule allows 15%). The pair where it has more is mysql-server. Its skill arm has no build to judge, and a run without one counts as worse than any number.

The table has two limits. First, the counts are of distinct causes, so a pair's comparison says nothing about how widely each cause spread. Second, a pair can tie while the causes differ, as qgis does.

Under the rule as first frozen, H1 is inconclusive. That rule counts every run without a whole build as worse, gdal's and xbmc's skill arms included, and keeps yugabyte-db's pair. It gives 6 fewer, 4 more and 4 tied. The pre-registration requires this reading to be reported beside the one in force. Amendments A12 and A13 are why the two differ (appendix C).

A16 moved 2 pairs. On gdal and cmake, the skill arm's migration switched assertions off in a way the parity measure does not record. A16 counts it as the build-type cause it is, and those pairs become ties. Before A16 the tally was 9 fewer, 1 more and 3 tied, also supported (appendix B).

5.3 H2: refuted, on the one run it applies to

Of the L and XL projects, only mysql-server's specification names a test driver other than ctest (mysql-test-run.pl). Its no-skill run neither wired that driver up nor ran it: neither its patch nor its transcript mentions it. H2 is refuted on 1 run, which says less than its label suggests.

5.4 H3: the tests that were ported pass

supported. All 5 of the skill arm's checkable runs are true, and 8 of 9 of the no-skill arm's. The false one is yugabyte-db's no-skill arm: its servers abort at startup, and 68.6% of its matched cases fail (§6.14; its tests run through a wrapper script, so each counts as one case). The skill arm's checkable runs are lensfun, pono, jansson, openttd and bitcoin. The skill arm ported none of the reference's test cases on any L or XL project where it produced a build. So H3's skill-arm half is a statement about small and medium projects.

5.5 H4: the arms point opposite ways

inconclusive. The skill arm changed the reference's CMake options in 2 of 2 XL runs and 5 of 8 at M and L. The no-skill arm changed them in 2 of 3 XL runs and 6 of 7 at M and L. The measure reads the agent's own CMake configuration, which says what it compared itself against, not how much of the project its Bazel build covers. The measure reads the latest CMake cache configured from the agent's source tree, in the workspace, beside it, or in the container's temporary directory. 2 runs left none and are left out: runs 060 and 071, mysql-server's skill arm and freecad's no-skill arm. yugabyte-db's skill arm, which configured CMake and then declined, counts as having kept the options. Source coverage (§6) is the better guide to how much each migration did.

5.6 H5: cost rises with size, in both arms

supported. Spearman's ρ between notional cost and the reference's distinct compiled sources is 0.714 in the skill arm and 0.820 in the no-skill arm, across all 14 pairs.

6. The fourteen pairs

One paragraph and one table per project. Each is written from the pair's two run records (runs/NNN-*/RESULTS.md) and cause tables; the tables are rendered from them. In the tables, "reference test cases ported" counts the reference's passing test cases that have a counterpart in the migration, and "pass" means at least 99% of those pass. A dash means the measure does not apply: no build, no test suite, or no generated files the lab could pair. A run judged on the part that builds (†) has source coverage and causes, which the lab can extract from the targets that build. It has no test or generated-file figures, because the lab's test and generated-file steps run on a whole build. Each arm is judged against its own run of the reference's test suite, so a flaky reference test can make the two arms' denominators differ slightly.

6.1 graphicsgems (M, 308 reference sources)

The happy path for the skill. Its migration compiles every reference source with no behavioral cause, and its agent kept the reference's CMake options. The no-skill arm's run replaced CMake rather than adding to it. It dropped -Werror, compiled with -w, and carries one behavioral cause, a code-generation flag (-fcommon). The project has no test suite.

6.2 lensfun (S, 40 reference sources)

Both arms compile every reference source and pass every reference test case. The skill's documentation lists lensfun's custom command as a place to stop, and the skill arm migrated it anyway, as a genrule. The no-skill arm's one behavioral cause is OpenMP applied to test binaries where the reference does not apply it.

6.3 pono (S, 94 reference sources)

The closest pair in the study. Both arms leave out the same dependency sources, which pono pins as external modules (a structural cause). Both pass every reference test case. Both compile at -O2 where the reference uses -O3, because both mapped CMake's Release to Bazel's opt (P18). Neither has a behavioral cause.

6.4 jansson (S, 34 reference sources)

The cleanest target. The skill arm's migration leaves no cause of any class, and every reference test case passes. The no-skill arm's also passes every case. Its remaining causes are structural or benign: it expresses jansson's configuration mechanism differently, and its linker options differ. None of its 2 generated headers is byte-identical to the reference's.

6.5 openttd (M, 535 reference sources)

Both arms build, pass every reference test case, and carry one behavioral cause in common. That cause is the stub CMake generates to compile a precompiled header, which neither migration compiles. The no-skill arm adds a second: it configured its own CMake comparison as Release, and its Bazel build compiles every unit at -O2 with assertions off, where the reference sets neither. Both arms handled a hazard the action graph cannot show. OpenTTD registers drivers through static constructors that nothing references, so an archive lets the linker drop them. The skill arm's agent found this when its converged binary failed at startup, and added alwayslink. The no-skill arm's agent added it while wiring up the build. Both binaries also report no version, because neither build hands the version generator git information (F32). The no-skill run cost 3.0× as much and took 2.8× as long, for the same coverage.

6.6 bitcoin (M, 548 reference sources)

Both arms leave out nearly the same reference sources. The no-skill arm compiled more of the project than its coverage shows, a benchmark suite the reference does not build. The skill arm's defines match the reference on every unit (V2). The no-skill arm has 3 more behavioral causes:

• a feature define it dropped (ENABLE_EMBEDDED_ASMAP);
• hardening flags it dropped;
• a Boost.MultiIndex configuration macro it omitted, which changes the code that library compiles to.

Both arms ported the same test binary, and its cases match 825 of 1,049 and 827 of 1,049 of the reference's cases, all passing. Neither ported secp256k1's or the IPC tests. Both compile with no optimization level where the reference uses -O2. Most of the skill arm's generated files differ from the reference's in bytes, by layout and by a -dirty suffix that its version script adds because the migrated checkout is modified. Most of the no-skill arm's are identical, and the rest differ only in layout (F33).

6.7 mysql-server (XL, 4,680 reference sources)

The skill arm's migration cannot be rebuilt from its own repository. Its agent said so itself: code generation was left to CMake, and the Bazel build consumes CMake's output directory as sources. That directory is not in the migration's repository, and holds links to CMake-built libraries that exist only after a CMake build. A rebuild from the migration's bundle fails to link (F35). In the lab's rebuild of the agent's own workspace, restored with CMake's output still in place, 581 of 585 declared targets build (the table's figure), and mysqld is among those that do not. This is the one pair where the skill arm was the slower and the more expensive: 103 min and $23.01, against 22 min and $14.56.

The no-skill arm wrote its own Starlark configure_file and migrated 9.4% of the reference's sources. Its other behavioral causes are real:

• the distribution's hardening and link-time optimization flags dropped;
• _FORTIFY_SOURCE at a lower level than the reference's;
• feature and vendored-library defines decided differently;
• configure checks that differ from the reference's.

It ported 2,350 of 11,385 of the reference's test cases, all passing: too few to be checkable, far more than none.

6.8 gdal (L, 1,554 reference sources)

The skill arm's whole build stops on one fuzz harness, where a driver-registration function is not declared in scope. With --keep_going, 280 of 283 of its declared targets build. Judged on that part (†), it compiles 92.7% of the reference's sources, against the no-skill arm's 93.6%. The two carry the same number of behavioral causes. One of the skill arm's is build-type: its part-build compiles every unit with -DNDEBUG, which the parity measure does not record, and amendment A16 counts (appendix B). Its agent also configured its own comparison with tests off and a Release build type (P13). The no-skill arm checked in a frozen cpl_config.h, the result of CMake's probe on the agent's machine. The configure check finds it differs from the reference's. The file carries a note to its maintainers to mirror any new feature check by hand (F39). It ported 1,215 of 1,229 of the reference's test cases, all passing.

6.9 xbmc (L, 1,871 reference sources)

Same image, opposite strategies, and the pair shows which one was a choice. Kodi's CMake downloads and builds FFmpeg, crossguid and libdvdnav for itself; the image does not carry them at the versions Kodi needs.

• The skill arm resolved every dependency from the host. It predicted before finishing that some units would not compile, and attributed that to the environment. Its own differ reported the migration converged, since a unit that never compiles still has a compile command to compare. Its whole build fails; 261 of 269 declared targets build, and kodi is not among them.
• The no-skill arm, in the same image, vendored those dependencies at the versions Kodi pins, with Kodi's patches applied, and built. So "the environment lacks it" was a strategy, not a constraint (F49, P23).

Judged on the part that builds, the skill arm covers 86.5% against 95.8%. It carries 3 behavioral causes to its pair's 7. Among the no-skill arm's is NDEBUG dropped, so its assertions are live where the reference compiles them out. The no-skill arm ported the suite as one test binary, and 1,687 of 1,689 reference cases match and pass. The skill arm's agent also ran a negative control on its own parity claim, unprompted: it planted differences, saw the differ report them, and restored them.

6.10 qgis (XL, 3,866 reference sources)

The skill arm's parity is a claim about a job it defined. It configured its own comparison with Python, 3D, WebKit, the tests and more turned off, and converged against that. Against the lab's reference, the cut is visible as absent sources and targets, feature defines decided differently, and the other causes in the table. The no-skill arm attempted more of the project: the core, analysis, GUI, 3D, server and application libraries, provider modules, and 1,981 of 4,905 of the reference's test cases, all passing (the no-skill arm's run of the reference suite passed one network test binary that the skill arm's did not, and that binary registers the 13 cases by which the denominators differ). It covers 72.2% against the skill arm's 69.2%. The two arms tie at 7 behavioral causes, but not the same ones. Both have a build-type cause, for different reasons: the skill arm added NDEBUG, and the no-skill arm dropped Qt's QT_NO_DEBUG, which turns Qt's own assertions on. The added NDEBUG is recorded here, unlike on gdal and cmake, because the same units already differ in other defines, and a migration-only define rides along into that cause (appendix B). Export macros that sources read are the skill arm's alone. -fvisibility=hidden dropped is the no-skill arm's alone. The skill arm's agent also found the differ's hash-order nondeterminism independently, and fixed its cause (P14).

6.11 freecad (L, 2,751 reference sources)

Both arms narrowed the job. The skill arm migrated FreeCAD's non-GUI core, 8.2% of the reference's sources, and named what it left out and why rather than dropping it silently. OpenCASCADE has no Bazel packaging, and PySide/shiboken, which generates the GUI bindings, has no rule. It compiles at -O3 where the reference sets no level, which parity does not see. Its one behavioral cause beyond the scope cut is build-type: it drops Qt's QT_NO_DEBUG, so Qt's assertions are on where the reference has them off. The no-skill arm went further, 26.1%, and by its own account produced an install tree that does real CAD work. It ported 1,257 of 1,440 of the reference's test cases, all passing. It carries more behavioral causes, including link-time optimization dropped and configure checks that differ. It cost 2.3× as much, for 3.2× the coverage.

6.12 cmake (L, 1,158 reference sources)

The same job in both arms. Coverage is 90.9% and 90.8%, and the two arms share two behavioral causes, absent targets and absent sources, and each adds one. The no-skill arm's: a few of CMake's C sources compile at -std=gnu11 where the reference uses -std=c11. The skill arm's is one the parity measure does not record: it pins -c opt, so every unit gets -O2 and -DNDEBUG where the reference sets neither. Amendment A16 counts that as build-type, and the pair ties (appendix B). The configure step was done two ways:

• The skill arm checked in all of CMake's configure-time headers, and each is byte-identical to the reference's.
• The no-skill arm regenerates most of them at build time from the templates. Its generated-file row counts a different, larger set of files, found automatically. So the two rows do not compare.

Neither arm ported CMake's test suite, which is itself made of CMake projects. The skill arm was faster and cheaper here, and in the replicate (§7).

6.13 blender (XL, 5,101 reference sources)

Both arms made the same scope decision, independently. Each built Blender's blender_lite preset rather than the default configuration, and they cover nearly the same share of the reference: 2,601 of 5,101 and 2,604 of 5,101 sources. Most of their behavioral causes follow from that preset: features off, their sources uncompiled, and assertions kept on in a release build, which the preset asks for.

The no-skill arm adds 3:

• a platform macro it omits;
• build-time generator tools compiled without the reference's -march;
• position-independent executables where the reference links everything -no-pie.

The reference builds with Blender's C++ unit tests off, so its passing tests are Python and whole-program tests. Neither arm ported any of those. The no-skill arm ported Blender's gtest suites, which the reference does not run, so there is nothing to match them against. The skill arm's agent found two things about the skill itself. The differ's verdict on its own final models changed with Python's hash seed, and it wrote a deterministic wrapper (P14). Its diff had converged while Blender's Python modules failed to load, for want of an export flag the parity model does not read (P25). The skill arm cost $26.52 and the no-skill arm $103.98.

6.14 yugabyte-db (L, 2,137 reference sources)

The skill arm declined, and was right. After 162 s and $1.19, its agent stopped with a report instead of a migration. It named the blockers, the hardest a PostgreSQL fork built by autotools outside the CMake model, which the servers link; the lab confirmed the reference model compiles nothing under src/postgres. The no-skill arm did the migration, and broke exactly there:

• In 95 min and $42.23 it migrated 84.6% of the reference's sources.
• Its bazel build //... passes in its own workspace, where CMake's outputs already exist; from its repository alone, CMake and the project's configure script must run first. The build links the CMake-built PostgreSQL backend, along with CMake-built copies of yugabyte's own libraries beside the Bazel-built ones. Its servers abort at startup, and most of its failing tests die on the same command-line flag registered twice, which is what two copies of one library do.
• Of 573 matched tests, 31.4% pass. Its tests run through a wrapper script, so each counts as one case.

The skill arm's report was an accurate forecast, at a small fraction of the cost, and it left nothing a user could mistake for a finished migration. The skill's instruction to stop at unsupported constructs was passed over in every earlier run it applied to (F12). The project ships its own coding-agent instructions, and they reached both arms. Their directive against running CMake directly did not prompt the decline: the skill arm never read the file it is in, and the no-skill arm read it and went ahead. Whether the rest of that configuration had any influence is not known (F53). The pair is left out of H1.

7. Repeated runs: how much one run can say

Every result in §5 and §6 rests on one run per arm per project. Two projects were run twice per arm, and S01 ran four small projects three times per arm. Both show how far a second run can move.

7.1 cmake, run twice under identical conditions

The pair was run again with the same commit, image, caps and byte-identical prompts (runs 075 and 076; A5).

What held. All four runs build, and coverage stayed within a few sources. Both runs of the skill arm have the same behavioral causes. The skill arm was faster and cheaper both times.

What moved.

• The margin. The no-skill arm cost 2.4× the skill arm's the first time and 1.3× the second.
• The no-skill arm's causes. Its two runs differ by one cause each way. The first compiles a few C sources at a different standard; the second switches assertions off, as both skill runs do (A16).
• The skill arm's configure headers. This is the one that matters most, and parity cannot see it. The skill arm checks in CMake's configure-time headers. In its first run all 28 match the reference byte for byte; in its second, 25 of 28. The 3 that differ are not cosmetic. The vendored curl is built with HTTP Strict Transport Security disabled. It and the vendored expat take a different random-number path than CMake's own probe found. libarchive is built without its OpenSSL-backed hashing, in an image that has OpenSSL (F51).

No test executes those headers, and the differ never reads them. The difference is on the record only because the lab byte-compares generated files, and between the two skill runs only because the replicate exists.

7.2 mysql-server, run twice, and the runs disagreed more than the arms

A pilot pair ran at the same commit, in the same image, before S02's rules were final (runs 038 and 039; a third pilot run, 037, is not counted: its session ended mid-migration when its agent backgrounded a build and ended its turn, F27). The pilot was judged before the comparable reading existed, and its tree cannot be rebuilt, because the registry snapshot it resolved against is gone. So it is compared only on measures that do not depend on the reading.

The skill arm built at 73.8% coverage in the pilot, and failed its whole build in the study run. The no-skill arm's two runs differ 10.5× in coverage, 14.0× in time and 5.0× in cost. Those spreads within an arm are larger than the difference between the arms on the same project (F41). On this project, how much an agent decided to finish varied more than anything the protocol controls. The pilot's registry snapshot can no longer be recovered, so module resolution may also have differed between the runs, and the spread may not be purely the agent's. No XL statement in this report should be read as more than one draw.

7.3 S01: four small projects, three runs per arm

S01 was the lab's first study. It ran graphicsgems, lensfun, pono and jansson three times per arm on the host, before the lab moved runs into containers. Its runs were first judged under S01's own rules. For this report they were re-judged under S02's comparable reading from their stored models, with every tree rebuilt so the checks could run (appendix C). Run on S02's own runs, which were judged this way from the start, the same procedure reproduces their error counts and cause lists exactly (V0880).

The skill arm was the more repeatable.

• Causes. It has no behavioral cause in 11 of its 12 runs. The exception is pono's run 025, which put the project's release settings behind a named --config=release. Judged at Bazel's default invocation, as every run is, its build type differs from the reference's. It is also the one S01 run whose program output differed from the reference's (F21).
• Cost. Its spread from cheapest to most expensive run (the ratio in the table) was narrower than the no-skill arm's on every project.
• The no-skill arm. Its causes repeat on lensfun (codegen-flags in every run) and vary on pono, where one run rebuilt every solver dependency from source at $51.97. That figure is a lower bound, because one background subagent's tokens were not stored (E23).

Three runs per cell (one project in one arm), on the host rather than in containers, is still not a replication of S02's conditions. S02's phase A re-ran the same four projects under its own conditions, and those are pairs in §6.

8. What the record says the skill should change

Each proposal below cites the findings behind it. The full text, with its evidence, is in SKILL-FEEDBACK.md. The lab has also implemented them, one proposal per commit, on a local branch of the any2bazel repository (lab/skill-feedback, not pushed), which no run in this report used. Re-run on the lab's stored models, it gives the same error counts, kind by kind, as the version tested; the NDEBUG change reports migration-only defines separately rather than changing those counts. Its handling of defines is also checked against the compiler's own preprocessor.

8.1 "Converged" means less than it reads

An agent that converges has matched the reference's compile commands, as the differ sees them. Several things fall outside that, and the skill does not say so where the agent reads the result:

8.2 What the agent compares against

8.3 The scope boundary

8.4 Defects in the skill's scripts

The remaining proposals (P5, P6, P7, P12) are marked medium or speculative in SKILL-FEEDBACK.md.

9. What the results support, and what they do not

10. Lessons for the next study

What S01 and S02 taught about running a study like this. Most lessons came from something going wrong first. The repository keeps each one where it was learned, in the runbook's warnings, the findings and the amendments.

10.1 Designing the comparison

• Pair every run, in the same image. xbmc's skill arm reported that the environment lacked three dependencies, and every clause of its report was true. Only its pair showed the conclusion was wrong: in the same image, the no-skill arm vendored them and built (§6.9, P23). An unpaired run would have recorded a constraint where there was a choice.
• Take the tool off the disk, and check afterwards. Deleting the skill's name from the prompt does not keep it out, because its description triggers on the request itself. The no-skill arm means the skill is not installed, and a check after every run confirms the agent never loaded it.
• Configure the reference before either arm runs, and judge against it. At XL, both arms of the pilot redefined the reference configuration, one turning off whole subsystems and the other the distribution's compiler flags, and said so only in their configuration files (F28). qgis's skill arm converged against a comparison it had narrowed (§6.10). A reference the lab fixed in advance is what makes "converged" and "covered" mean the same thing in both arms.
• Pre-register repeated runs. One run per project and arm was frozen into S02 knowing S01's spread on small projects, but not how far a run could differ at XL. On mysql-server, two runs of one arm differed more than the arms did (F41). The design should fix the number of repeats per project and arm, and the analysis should use the spread, from the start.
• Unattended runs change the tool being tested. In non-interactive mode the skill's confirmation question goes unanswered and the agent carries on (F2). An agent that backgrounds a long build and ends its turn ends the session (F27). And the skill's instruction to stop at unsupported constructs was passed over in every applicable run until yugabyte-db (F12). A study of an interactive tool run unattended measures the unattended tool, and should say so.

10.2 Measuring

• The tool's own instrument cannot be the only instrument. Parity was the right thing to check, because it is what the skill promises, and it missed things. It missed what generated files contain (F1), shared-library link lines (F14), optimization level (F30), flags and defines only the migration adds (F31, F56), and whether the program runs (P25). Each was found by a measure that did not come from the differ.
• Count decisions, not errors. An error count measures how widely one decision spread. It moved several-fold between two runs of the same arm on cmake with almost no change in what was built (F51). The path map alone moved it by more than any decision weighed (F47). Causes, one per decision, are what compare.
• Fix the units before the first run. Two measures had to be corrected because their unit was wrong, one during the study and one after its analysis:
  • Scope was first read from compile actions, which count a source compiled twice as two, and it overstated coverage (F40).
  • Tests were counted by CMake's registrations, which are one case on one project and a merged binary on another, and ported tests read as not ported (A15, F57).
  Both units look reasonable until a project arrives that breaks them.
• Prove the instrument against itself. The differ's verdict depended on Python's hash seed wherever a source was compiled in several variants (P14). A migrating agent reported it first, in the mysql-server pilot, and the lab confirmed it in its own judge. Two later agents, on qgis and blender, found it independently and worked around it. Every harness should be run against itself, and should report identical, before any of its numbers is believed.
• A guard that has never been seen to fire is not known to work. One of the lab's safety guards could not fire at all, and another was guarding a path that would have deleted the container runtime's own files (F52). Test the guard by making it fire.

10.3 Recording

• Verify from artifacts, not summaries. Agents' own reports were largely accurate, and contained stale claims in the same paragraph (F9). When the artifact is computed (a rendered page, a script-filled table, a check), reading its source is not reading the artifact: run it. A report once passed two link audits, and rendering it found broken links that existed only after its scripts ran.
• Keep the literal record. Every verification in the lab is a command and its output, kept in full (with any truncation flagged) and cited by id (the V numbers in this report). A sentence saying something was checked reads the same whether or not it was.
• Correct a wrong measure openly. When a measure is found to describe migrations wrongly after the analysis, the lab corrects it as a dated amendment, states which way the correction moves the result, and keeps the uncorrected result beside it (A15, A16; appendix C). Corrections to records are indexed as errata, with a marker at the corrected spot, so a reader can calibrate how far to trust the rest.

10.4 Writing it up

• One reading per report. The records support several readings: which runs count, how a run with no build is scored, which configuration is comparable. A report that states several at once invites sentences that mix them. An earlier draft of this report did. Rounds of review kept finding a steady share of statements that did not hold, many of them numbers true under one reading and quoted beside numbers true under another.
• Render the numbers from the records. Every number here is rendered from a file generated from the run records, and the build refuses a typed one. That removes transcription errors. It cannot stop a number rendered from the wrong fact, and two such slips were caught in review.
• Review with two kinds of reader. A reader given only the document finds what it fails to teach in the order it is read. A fact-checker given the repository finds what the document gets wrong. Each finds problems the other mostly does not, and every finding of either should be checked against the record before it is acted on (reports/v2/REVIEW.md).

Appendix A. Every run, and how to reproduce the figures

Every run number the lab has allocated. A number is never reused. An abandoned run keeps its number, with the reason in its manifest. "In this report" names the section that uses the run. The others are shakedowns, pilots and validation runs, under conditions this report does not compare. S02's phase A re-ran S01's four small projects under S02's conditions; its phase B ran the ten larger ones.

Reproducing the figures. From the repository root:

python3 studies/S02-large/analysis/primary.py      # the verdicts (writes primary.json, PRIMARY.md)
python3 reports/v2/facts.py                         # every figure in this report (writes facts.json)
python3 reports/v2/build.py                         # this page; refuses any number not rendered from facts.json

The per-run records these read are in runs/NNN-*/:

• RESULTS.md, the run's record;
• judge/, the lab's measurements;
• run/, the launch, the transcript and the token counts.

Each migration is stored as a git bundle against its upstream commit. Findings (F), proposals (P), errata (E), amendments (A) and verification entries (V) are in the repository's FINDINGS.md, SKILL-FEEDBACK.md, ERRATA.md, studies/S02-large/PREREGISTRATION.md and verification/.

Appendix B. Assertions, and amendment A16

This appendix explains a rule the body applies. Parity cannot see a migration that switches assertions off, unless another define error on the same units carries it along. Amendment A16 counts it anyway, reading the builds directly. The pre-A16 tally is kept in appendix C.

B.1 The blind spot

Parity compares the compile flags of each unit the two builds share. A define the reference has and the migration lacks is an error, and every error is attributed to a cause. A define that only the migration has is a warning, and causes are built from errors. So a migration that compiles with -DNDEBUG against a reference that does not is recorded as nothing at all. The exception is a unit that already has another define error: there the extra define rides along into that cause. NDEBUG is the define that turns assert off. Whether a build has assertions on is exactly the kind of difference the behavioral class exists to catch.

B.2 Which runs it touches

Among the 14 pairs, the migration's assertion state differs from the reference's in 8 runs. A build's state here is the one most of its compile actions have. Of those runs, three are mixed: qgis's skill arm compiles 98.7% of its actions with NDEBUG, yugabyte-db's no-skill arm 18.9%, and xbmc's no-skill arm 0.1%, in one build-time generator tool.

Turning assertions on removes a reference define, so it is always an error and always seen. Turning them off adds one, and is seen only by accident. The skill arm turned them off 3 times and on 1; the no-skill arm turned them off 1 and on 3. So the blind spot is not neutral between the arms. Parity saw 4 of the no-skill arm's 4 assertion changes and 2 of the skill arm's 4.

The two unseen cases came about differently. cmake's skill arm mapped CMake's build type to Bazel's --compilation_mode=opt, which brings -O2 -g0 -DNDEBUG whole, where the reference sets none of them (P18). gdal's skill arm passes -DNDEBUG itself, mirroring a Release build it configured for its own comparison.

B.3 What A16 does

A16 gives a run the behavioral cause build-type when three things hold. Its Bazel build defines NDEBUG on units the differ paired with the reference's. The reference does not define it. And no cause already records it. The runs it applies to: 062, 073, 075, 076.

gdal and cmake move from "fewer behavioral causes with the skill" to ties. The verdict holds. Counting assertion state alone, seen or not, the arms are even.

B.4 Why it is an amendment

The blind spot is a flaw in the measure, not a different reading of it, and it runs in one direction: it credits whichever arm prefers release defaults. Keeping it would have kept a tally known to describe two migrations wrongly. So A16 was adopted after the analysis, although its effect was already known, which is recorded with the amendment. It covers NDEBUG only, because its effect is known; other migration-only defines have not been classified one by one. The general fix belongs in the differ (P17): treat a migration-only define as an error, from the next study on.

Appendix C. The other readings the records support

C.1 H1 under the rule as first frozen

As first frozen, H1 counts every run without a whole build as worse than any number of causes, and keeps every pair. Amendment A12 judged gdal's and xbmc's skill arms on the part that builds. A13 made that the primary reading and left out yugabyte-db's pair, whose skill arm declined. Under the frozen rule: fewer 6, more 4, tied 4, which is inconclusive. The pre-registration requires this reading to be reported, and §5.2 does, in one sentence.

C.2 H1 before amendment A16

Before A16, a migration that switched assertions off without the parity measure recording it had no cause for it. H1's primary reading was then fewer 9, more 1, tied 3, supported. The frozen-rule reading was fewer 7, more 4, tied 3, inconclusive. A16 moved gdal and cmake from the skill arm's column to ties (appendix B).

C.3 H3 before amendment A15

Before A15, tests were counted by CMake's test registrations rather than by case. Registrations are one case on some projects and a whole suite or merged binary on others. That reading made bitcoin's two arms and freecad's no-skill arm non-checkable, although their ported tests pass. H3 was supported either way: skill 4 of 4, no-skill 6 of 7 checkable runs true, against 5 of 5 and 8 of 9 under A15. A15 was made after the analysis of record, because the earlier count described ported tests as not ported (F57, E29). PRIMARY.md gives A15's H3 as the result and the earlier one beside it.

C.4 S01 as S01 judged it

S01's runs were originally judged under S01's own policy. The skill arm was judged against its agent's own configuration, including the suppressions the agent chose. The no-skill arm was judged against a map the lab authored. Bazel was queried with any build flags the agent declared. For this report every S01 run was re-judged under S02's comparable reading (§4.2): the 24 runs of its second phase, which are the ones this report uses. This used the stored models, rebuilt trees for the checks, and, for runs 024 and 025, Bazel's default invocation. The original records are unchanged beside the new ones (judge/s02-reading/; audits/s01-rejudge/).

The runs whose behavioral causes change:

• Run 025. Under S01's policy its release settings, behind the agent's own --config=release, were part of what was compared. At the default invocation, its build type differs from the reference's, the same difference its program's output showed (F21).
• Run 029. Its checks could not run when S01 judged it, so 3 causes stayed behavioral. With the tree rebuilt, the checks ran and found none of them to be behavioral.
• Runs 017 and 021. Under S02's reading they gain causes the S01 policy did not count: absent sources and, for 017, executables that are not position-independent where the reference's are.

C.5 Error counts

Every run's record carries raw error counts under several readings. This report uses none of them, because a count measures how widely one decision spread, and it depends on a path map that is not the same for both arms (§4.2, F47). On cmake, the map alone moves the skill arm's count by more than any decision in either migration weighs (F47). And within one arm, cmake's no-skill count moved several-fold between two runs whose coverage is nearly the same, mostly because of one include declaration (F51, which also records the configure-header change in §7.1).