Why Tests Became the Product: A Conversation with Titus Winters

A conversation with Titus Winters · Notes From The Fleet series

Watch the full interview

I sat down with Titus Winters to talk about developer experience and infrastructure at scale. Titus co-wrote Software Engineering at Google; he is now a Senior Principal Scientist at Adobe. I expected a conversation about build systems. What I got was a line I've been thinking about ever since:

"The question is not what can you build with an incredibly fast, 80% accurate tool. It's what can't you build with an infinite army of incredibly fast, 80% accurate tools."

An infinite army of 80% accurate tools. That army doesn't have a code generation problem. It has a verification problem. Whether it helps you or buries you depends on the systems that check, integrate, and maintain everything it produces.

Titus Winters interview

Titus Winters on the shift from code generation to verification in the AI era

Introducing any2bazel - an OSS tool for agent-driven migrations to Bazel

Back in June I was sitting on a train to Zurich. The plan was to meet colleagues and work together on figuring out how to migrate CMake projects to Bazel. I was looking forward to chatting with my colleagues in person, but I was not optimistic about the migration process. Truth be told, I was expecting to spend the week walking through the problems parsing the custom CMake language and CMake being Turing complete, maybe even look at a few concrete cases like generated files and configuration flags, finally conclude it couldn't be done, and return home with not much to show. Oh boy was I wrong.

  • in engflow
  • 4 min read

Over 3 Billion Actions per Week: A Year Inside the World's Largest Bazel Fleet

The EngFlow fleet just crossed 3.5 billion actions processed per week. Fourteen months ago, that number was 250 million. That's roughly 14x growth in 14 months.

This isn't a vanity metric. It's a signal of the developer efficiency our platform enables, and of the scale at which we now operate, managing some of the world's most complex and demanding platform engineering workflows for enterprises including ARM, Asana, Block, BMW, Canva, Databricks, Plaid, and Zoox.

14x growth in build actions

14x growth in 14 months in build actions per week across the EngFlow fleet.

Four War Stories and a Demo: Seattle Build Meetup 2026

Waleed Khan from Stripe presents in Seattle at Uber’s offices

On July 28, 2026, Uber and EngFlow co-hosted a Build Meetup at Uber’s Seattle office. This wouldn’t have been possible without the help of Sergey Balabanov at Uber, who not only presented one of the talks, but also energetically arranged and helped support this event from the Uber side. Thanks again, Sergey!

And, even though nobody planned a theme for this particular meetup (our meetups are holistically focused on “Build, Scale, Investigate”), one showed up anyway. Code is getting cheaper to produce. Uber put it flatly: “AI is a problem, apparently.” Every talk examined some part of the machinery that turns abundant code into software you can trust. Let’s take a look!

From Prototype to Production: The EngFlow Build Analytics Journey

Every build tells a story. Which targets took the longest? Where did cache misses cost you minutes? Who triggered the invocation, and what source changes drove it? At EngFlow, we believe that surfacing these answers — automatically and at scale — transforms build optimization from reactive firefighting into proactive engineering. Today, we're sharing how our build analytics platform evolved from early prototypes to a production-grade system providing tangible value to our customers.

Say hello to the new bazel.build website

Today, the new https://bazel.build/ website has finally launched. This is the first step in a larger evolution of Bazel's documentation story. Much of the content remains the same (the eagle-eyed among you may notice a few new pages), but this change marks a new era of Bazel documentation.

What motivated this change?

The Bazel site migration is more than just a fresh coat of paint. One of the underlying motivations for this change is to empower the Bazel community to make more changes to Bazel's documentation. Before the migration, Bazel's website was built on Google documentation infrastructure. While the capabilities of that infrastructure were largely sufficient for Bazel's needs up to now, the contribution experience was not.

Today's https://bazel.build/ is built on Mintlify. With this new platform, each incoming documentation pull request to Bazel is automatically greeted with a preview. For even faster iteration, you can build the site locally. These improvements eliminate guesswork, and better equip contributors to make changes to the Bazel docs with confidence.

Iterable cuts P90 build time from 42 minutes to 17 minutes with EngFlow

By migrating its Scala backend from sbt to Bazel with EngFlow and VirtusLab, Iterable created a faster, more reliable build foundation for AI-era engineering velocity.

42 → 17 minutes
P90 build time

~3 minutes
Common build time for smaller changes

~97%
Integration-test pass rate on master, up from the 70s

Iterable is an AI customer engagement platform trusted by global brands to create personalized experiences across every channel. Behind the platform is a distributed engineering organization building and operating a large, complex software estate.

For Iterable’s Developer Platform team, the mission is straightforward: help developers release software faster, with greater reliability and safety.

Build Forensics: How we uncovered a Rust HTTP/2 bug

About 5 months ago, one of our customers began reporting that 5% of their jobs based on Meta's Buck2 build system were failing. Even though there was no simple way to reproduce the phenomenon, all signs pointed to network hangs when a Buck2 client made API calls to their EngFlow cluster. If a simple timeout was added, the request would quickly time out, potentially indicating the issue was a bug in our remote execution platform.

Because of the potential for it to expose a broader issue in our software, especially in terms of compatibility issues with Buck2, we elevated the urgency of this investigation. After examining a few thread dumps that revealed nothing conclusive, we eventually scheduled a call with the customer. After performing multiple tests, it became evident that our schedulers were simply idle. If it were truly a problem in our system, we were unable to find evidence.

The Case of the Irregular Hanging: A build forensics story from the front lines of remote execution

There is a particular kind of failure that senior infrastructure engineers recognize immediately. Nothing crashes. Nothing spikes. Nothing logs an error. And yet… work simply stops.

This is the story of one of those failures and why Build Forensics matters once build systems become production infrastructure. For an in-depth technical investigation see this engineering blog.

Reduce LLVM Build Artifact Storage Costs by 50% with Content-Defined Chunking

One of the classic strategies to speed up a system is to avoid redundant work. Scalable build systems like Bazel and Buck2 heavily employ this strategy in various ways, with remote caching and content-addressable storage being two prominent examples. While remote caching prevents repeating redundant build actions, content-addressable storage (CAS) exists for the purpose of data deduplication. However, traditionally CAS operates at the granularity of a single file. When you modify a single byte in a file and store it in a CAS, the CAS stores a second, complete file. Deduplication at the file level is quite palatable for smaller files; a single build invocation typically contains many thousands of small files. Who cares if we store a couple more?

However, as file sizes increase, the cost of storing yet another slightly modified version of a file becomes more expensive. Instead of tossing a couple extra kilobytes into storage, you might be storing a few more gigabytes. Now, consider where these large files come from. These large files are often outputs of build actions, and those build actions depend on many smaller inputs. As those many inputs churn, the outputs also churn. Suddenly the cost of touching a tiny little source file isn’t just the cost of uploading a new version of the source file to CAS–it’s now also the cost of all the large outputs that are produced by the build. This dictates the growth rate of your CAS storage costs, which scales directly with the number of incoming builds. At AI-scale, these costs have become more important than ever before.