There is a particular kind of failure that senior infrastructure engineers recognize immediately. Nothing crashes. Nothing spikes. Nothing logs an error. And yet… work simply stops.
This is the story of one of those failures and why Build Forensics matters once build systems become production infrastructure. For an in-depth technical investigation see this engineering blog.
Imagine reading this post on your internal Slack:
Please hold off on pushing your PRs for a bit until the patch release is fully cut. We don't have enough capacity to handle all the load and the patch is blocked because of that.
Ouch.
But let's be real: not all builds are made equal - some builds are more urgent than others. Like here, when creating a patch release, and wanting to expedite the builds required for it over others.
Yet there's got to be a better solution than asking everyone else to not push changes.
The main service we provide at EngFlow is Remote Execution (or RE for short). Remote Execution allows a build to run over a horizontally-scalable distributed system, thus speeding up highly parallelized builds. This parallelization is one of the features that make Bazel builds so scalable. Outside of Bazel, Buck2 and a few exotic build systems such as Chromium and AOSP, the adoption of this protocol is almost non-existent. Here at EngFlow we saw many C and C++ codebases successfully adopting RE, and decided to partner with tipi.build, experts in that field, to bring CMake support to EngFlow.