Why Tests Became the Product: A Conversation with Titus Winters¶
A conversation with Titus Winters · Notes From The Fleet series
I sat down with Titus Winters to talk about developer experience and infrastructure at scale. Titus co-wrote Software Engineering at Google; he is now a Senior Principal Scientist at Adobe. I expected a conversation about build systems. What I got was a line I've been thinking about ever since:
"The question is not what can you build with an incredibly fast, 80% accurate tool. It's what can't you build with an infinite army of incredibly fast, 80% accurate tools."
An infinite army of 80% accurate tools. That army doesn't have a code generation problem. It has a verification problem. Whether it helps you or buries you depends on the systems that check, integrate, and maintain everything it produces.
On record: Titus is no longer an AI skeptic¶
Titus was a skeptic until last winter, when the model improvements turned so many skeptics around. What's interesting is what actually turned him. The moment the game changed, in his words, "from hype around generation to some thoughtfulness around maintainability and review and correctness."
We're seeing the same shift with our customers: as code generation becomes easier, cheaper, and more voluminous, verification is the next frontier. This is the new opportunity for companies to innovate and build better tools and skills to help AI models use deterministic data about the build graph and build/test performance to feed into the model for better accuracy.
The break-even shift¶
My favorite piece of evidence in the whole conversation is an exercise Titus runs in an internal class at Adobe. He asks a room of team leads: how long does a project have to live before a practice pays for itself? Version control has always polled at two or three days. Code review, three or four. A year ago, unit testing polled at about three weeks.
This year, the same exercise returns three or four days. Think about that. Nothing about testing changed between those two polls. What changed is who writes the code and tests, and how easy it is to reach a higher test coverage.
"If you don't have that deterministic safety net in place, the bots just mess things up."
There's also a simple economics here. Without tests, the AI has to re-ingest your code and reason through whether a change is safe — every time, for every change. "Tests just give you a faster, deterministic, reliable way to do that cheaper."
Adam Bender made the demand-side argument in his Google I/O talk, Software Engineering at the Tipping Point: dependency graphs grow quadratically, not linearly, so a 10x larger code base can mean 100x more test load. In AI-scale development, the generated code is the cheap part. The test suite is the product and it requires a robust and scalable infrastructure to handle the test growth.
What this does to infrastructure¶
When I joined Google as part of the Bazel team, just as Bazel was open-sourced, what won me over was fast and correct: incremental builds, and you don't do clean builds—ever. For years that was a developer experience argument. Now it's a cost and capacity argument, because everything downstream of the tests absorbs the new load. Titus put a number on the stakes:
"If you're doing ten times the volume of code, you had better have ten times less maintenance cost per line of code. Otherwise, you are accelerating yourself into a crisis."
He also named the failure mode I worry about most: nobody gets paged when CI gets slow. And CI is about to become the bottleneck that decides whether companies recoup their AI investments.
We see this load arriving in our own fleet, and help our customers seamlessly absorb it with our platform and 24x7 support - we carry the pager for CI issues! On a day-to-day basis we don't feel it, until we pull the data. A year ago, our customers were running 250 million build and test actions a week. Today we're handling three and a half billion, and the growth rate is accelerating. Our customers keep absorbing more load.
What this means for you¶
If you're running an engineering org in 2026:
- Audit your build system as if it were production infrastructure, because it is. If you're running
make cleanor dealing with non-deterministic builds, you're paying a tax on every AI-assisted change your team makes. - Measure CI latency with the same rigor you measure API latency. Your on-call engineers watch dashboards for request times. Do you have a dashboard for build and test times? P50, P95, P99 by team, by time of day, by change size?
- Calculate the break-even point for testing in your org. If it's still measured in weeks, you're not going to keep up. If you don't know what it is, you're flying blind.
- Plan for 10x code volume with 10x less maintenance cost per line. Titus's rule isn't aspirational. It's table stakes. If you can't maintain code at 1/10th the cost per line, you can't afford to let your team scale output with AI.
The companies that will win the next phase of AI-driven development aren't the ones with the biggest AI budgets. They're the ones whose infrastructure can absorb 10x commit velocity without collapsing and whose builds are fast enough that CI doesn't become the bottleneck.
Check on your own cluster¶
Two numbers I'm curious about: what share of your weekly actions are tests rather than builds, and what are your CI queue times? The first tells you whether tests are already becoming your product. The second tells you what your bots are waiting for.
And the question I want answered: when your commit volume doubles again (and it will) which system hits its limit first: code review, CI, or the build itself?
Somewhere in the episode, I asked Titus about the Mythical Agent Month, what happens to Brooks's law when the agents work nonstop. His answer is at 44:42, and the full conversation is 52 minutes. It's worth all of it. Thank you, Titus!
Next in the series, I am going to share insights from the conversation with James Governor at Redmonk on why Bazel is having its moment in 2026 and our open source efforts to make it easier to benefit from.