From Prototype to Production: The EngFlow Build Analytics Journey¶
Every build tells a story. Which targets took the longest? Where did cache misses cost you minutes? Who triggered the invocation, and what source changes drove it? At EngFlow, we believe that surfacing these answers — automatically and at scale — transforms build optimization from reactive firefighting into proactive engineering. Today, we're sharing how our build analytics platform evolved from early prototypes to a production-grade system providing tangible value to our customers.
Validating the solution at scale¶
Our analytics journey began with a question: what if we could give engineering teams a complete picture of their build performance over time — not just per-invocation, but across their entire organization?
Early prototypes explored lake-centric pipelines and post-completion processing to answer that question. These explorations validated the concept and proved there was real customer value to be gained from organizational build insights. But they also exposed hard architectural limits. On our largest clusters, the memory pressure on the pipeline and performance bottlenecks on our core infrastructure made it clear that the architecture needed to change.
One problem crystallized the issue — requester identity. We tried to enrich the analytics data with data from the BES stream, but doing so caused instability in our client-facing APIs. To capture this data without degrading performance, we moved analytics processing onto the analyzers.
As our customers scale up to tens or hundreds of millions of build actions per week, scaling build analytics becomes essential.
Introducing enhanced build analytics at scale¶
Enhanced build analytics is a ground-up re-platforming onto EngFlow's analyzer infrastructure. Rather than processing build data during RE API execution, analytics are now written after the fact in a separate pipeline.
Here's what that re-architecture delivers:
Speed and stability at scale¶
By co-locating analytics processing with build analysis, we eliminated the memory and performance issues that plagued the original pipeline on high-volume clusters. The new pipeline handles the throughput of our largest customers without breaking a sweat. Builds complete, data lands — no lag, no backpressure.
Richer data, automatically¶
Enhanced build analytics doesn't just fix the plumbing — it brings materially better data. Source control metadata and build context are now captured alongside execution data, giving you the full picture of what changed, who built it, and how it performed.
Full observability at scale¶
We instrumented the analytics pipeline with Prometheus metrics covering queue depth, event age, processing volume, and other performance metrics backed by dashboards for operational visibility. When something needs attention, we know before you do.
Early signals¶
Customers using build analytics are already seeing significant improvements:
- One team reduced their metric ingestion time from 26 hours to 20 minutes — turning what was an overnight batch process into near-real-time insight.
- Another saw a 50% reduction in p90 build times by using the data to pinpoint and fix bottlenecks.
- Teams are using the platform to identify performance regressions and reallocate resources proactively — with much more to come as adoption grows.
Coming down the pipeline¶
The analytics platform is built to grow, and here's what's next:
- Dependency graph and critical path analysis: Understand not just which actions were slow, but which ones blocked everything else. Critical path visibility turns build optimization from guesswork into precision engineering.
- Per-target views: Drill into individual targets to track their build performance over time and catch regressions early.
Get started¶
Enhanced build analytics is available now for EngFlow customers. Want to see how your builds perform over time, identify optimization opportunities, and give your platform team the data they need? Get in touch with us, and check out our documentation.