Four War Stories and a Demo: Seattle Build Meetup 2026¶
On July 28, 2026, Uber and EngFlow co-hosted a Build Meetup at Uber’s Seattle office. This wouldn’t have been possible without the help of Sergey Balabanov at Uber, who not only presented one of the talks, but also energetically arranged and helped support this event from the Uber side. Thanks again, Sergey!
And, even though nobody planned a theme for this particular meetup (our meetups are holistically focused on “Build, Scale, Investigate”), one showed up anyway. Code is getting cheaper to produce. Uber put it flatly: “AI is a problem, apparently.” Every talk examined some part of the machinery that turns abundant code into software you can trust. Let’s take a look!
Tango: Computing Exactly What Changed in Uber's 3-Million-Target Bazel Monorepo¶
By Xiaoyang Tan, Uber
Uber’s monorepo has three million Bazel targets. Its old change-target computation tool ran a full Bazel query at two revisions (base and patched), hashed both target graphs, and compared them. It was correct, but every CI consumer paid for the entire graph twice, and none of the work was shared.
Tango, a.k.a. Target Analyzer in Go, was recently open-sourced by Uber, turning that computation into a service so target graphs can be cached and reused. The cache is keyed by the Git tree hash rather than the commit SHA; the tree hash represents content, so different commit histories that materialize to the same tree can reuse the same graph.
Tango also returns each changed target with its dependency distance from the nearest direct change. That hands policy back to the client: one workflow might test everything within one dependency hop; another might go farther.
Check out Tango here: https://github.com/uber/tango
CASFS: The Filesystem Behind GM's Remote Builds — a Terabyte Checkout in 10 Seconds¶
By Seth Bunce, General Motors
After Google discontinued the remote-build service Cruise had been using, Seth’s team built its own remote-execution system, content-addressable store, and FUSE filesystem. General Motors now uses that system for CI, Unreal Engine-based AV simulation, model training, and AI agents that run full Bazel builds and submit pull requests to humans—although, Seth said, “some stuff [is] automatically merging now too, which is kind of scary.”
The reported scale is enormous: tens of thousands of client VMs, tens of millions of requests per second, and terabytes of egress per second.
CASFS mounts a source tree immediately, then retrieves and warms content to local NVMe as it is needed. That lets compute begin against a terabyte-scale checkout in roughly ten seconds—not because CASFS transfers a terabyte in ten seconds, but because it does not wait for the entire tree to arrive. Seth reported that average build duration fell by 75%, a 4x speedup.
A Professional Git Hater Comments on the Commit-Graph¶
By Waleed Khan, Stripe
Waleed is a self-described professional Git hater, and focused his indictment on one operation: merge-base. It sits beneath three-way merging, history traversal, and fetch negotiation.
Git’s commit-graph makes graph walking much faster in practice, but merge-base remains O(n) in the worst case. Waleed’s critique was that its rows contain data that merge-base doesn’t need, parent traversal produces cache-unfriendly jumps, and generation numbers reduce the work without changing the worst-case complexity.
Sapling exploits the fact that monorepo history is mostly linear. Runs of single-parent commits are compressed into segments, so graph-query cost scales largely with branching and merges, rather than with every commit. A benchmark Waleed presented completed an ancestor query across years of Linux history in about 0.1 milliseconds, versus about 10 milliseconds for the older Mercurial implementation. Its IndexedLog storage is append-oriented and does not require periodic repacking to maintain performance.
The lesson: Git’s data structures are choices, not physics.
Squeeze More Juice: Overcommitting CI Machines at Uber — and Surviving the Consequences¶
By Sergey Balabanov, Uber
AI has made code changes cheap and plentiful, and Uber validates each change across its monorepo. Meanwhile, machines sized for peak memory demand spend much of their lives underused: Uber measured roughly 12% aggregate CPU utilization across weekends and off-peak hours.
Uber therefore overcommits CI jobs per machine, betting that their resource peaks will rarely overlap. The first safety net is a Bazel wrapper that retries OOM-killed builds on the same machine, reusing the warm build state.
Then came the war story. Some jobs timed out after 30 minutes but completed in under two minutes when retried. The failures appeared only when multiple Go link actions ran concurrently. Heavy disk activity initially suggested I/O contention, but the root cause was memory pressure.
The Go linker memory-maps its archives. Under pressure, Linux can evict those file-backed, reclaimable pages rather than invoke the OOM killer. When the linker needs them again, page faults pull them back from disk—potentially repeatedly. The result is a page-fault storm: memory pressure wearing an I/O costume.
Uber’s fix is Paralink, a small Rust gate integrated into the Bazel toolchain. It watches page-fault counters and delays new Go link processes when pressure rises. Sergey reported that overcommitment and its safety nets save 30–35% of CI capacity—thousands of machines.
Check out Paralink here: https://github.com/paralink-network
How to Triage and Optimize Bazel Builds with Claude, Build Analytics, and EngFlow MCP¶
By Jorge Acosta, EngFlow
When you decide to optimize your builds, where do you look first? Jorge Acosta, product manager at EngFlow, starts with the question underneath build optimization: what does your build actually look like? The execution-log graph, the critical path, and drag, the time you'd get back if a node disappeared, and more; because at current monorepo scale, you cannot expect to optimize CI times by investigating invocations one at a time.
In a live demo using Build Analytics, EngFlow MCP and Claude Code, Jorge retrieved the cluster’s slowest invocations and combined client-side Bazel profile data with EngFlow’s server-side remote-execution data in Claude Code using Engflow MCP. This produced a ranked list of the slowest builds, including relevant metadata.
Then, he went deeper to find the root cause of the slowness and potential fixes. This meant switching to a fully deterministic approach using EngFlow’s Build Analytics queryable data store to identify regressions (using Median Absolute deviations) ranked by the ones taking the most cloud compute time. These are the most expensive regressions, and fixing them provides the highest ROI.
Jorge then applied the same logic to identify performance gains, concluding with a comprehensive build health report that tracks regressions, improvements, and historical costs by instance type and Mnemonic.
This is how infrastructure teams demonstrate their value and fund the next optimization project.
The Pattern: Generation Got Cheap, Verification Didn’t¶
AI did not create every problem described at the meetup. It is increasing the load on systems already straining under monorepo scale. Change detection, CI economics, storage, observability, and even version control are being reworked around operating assumptions their original designers never faced.
That is Build Management at AI Scale—and it is where these organizations deploy some of their smartest people.
Interested in co-hosting a meetup with us? Want to suggest a location? Learn where we are going next? Visit meetup.build for more details.
Questions about Bazel? We can help. Contact an EngFlow build expert today!