Skip to content

leadership

The Burst Demand Crisis

I spend a lot of time with engineering leaders and developer infrastructure teams. Over the past six months, a pattern has been emerging: build queues are experiencing demand in bursts, not waves.

Last week alone, conversations in Sydney, Chicago, Los Angeles, and San Francisco all surfaced the same story:

  • Sunday: A customer running EngFlow Bazel RBE at 100,000+ cores wants to triple capacity over the next year. PR volume is surging with no sign of slowing.
  • Monday: Prospective customer's CI queue is in crisis. Engineering attention diverted from product work. First-time inquiry about RBE.
  • Tuesday: Existing customer tried sharding load across more CI workers. Result: higher cloud costs, same bottleneck. Needs help making workloads RBE-compatible.
  • Thursday: A large enterprise customer skipped our customer dinner - they were occupied with urgent internal testing, preparing to double their RBE load.
  • Friday: Reviewed load planning with a customer preparing for an AI generation hackathon - forecasting what their infrastructure must absorb in the coming months.

What’s common between GCP Next and the Sphere?

My impressions and insights based on attending the GCP Next event in Las Vegas on April 9-11, 2024.

GCP Next

Summary

Thanks to our Google Cloud Platform partner, I was able to attend the GCP Next conference in Las Vegas this year. This was my 2nd GCP Next; the first was in 2018 in San Francisco, when I was actually on the inside as part of the Google Cloud team, leading content for the DevOps tooling track!

The schedule and setting made it very clear that Google’s top goal for GCP Next this year was to promote AI, followed by security. There was also a prominent display of data and security platforms, but AI dominated even those vendor displays and presentations.

Bazel scales more than just builds

Bazel creates a structured layer between the details of building and testing for individual languages and the users, both human and machine, that perform build operations. This abstraction simplifies workflows and is the foundation for powerful build-adjacent systems not possible with other build tools. Scalable Bazel builds underpin a scalable organization and development culture.