Skip to content

The Case of the Irregular Hanging: A build forensics story from the front lines of remote execution

There is a particular kind of failure that senior infrastructure engineers recognize immediately. Nothing crashes. Nothing spikes. Nothing logs an error. And yet… work simply stops.

This is the story of one of those failures and why Build Forensics matters once build systems become production infrastructure.

The first clue: a stubborn 5%

About five months ago, one of our customers reported an unsettling issue. Roughly 5% of their jobs, running on Meta’s Buck2 build system and executing against their EngFlow cluster, were hanging indefinitely.

Not failing. Hanging.

There was no reliable reproduction. No obvious correlation with workload size, time of day, or cluster load. Adding a timeout caused the requests to fail quickly, which made the most worrying hypothesis unavoidable:

Is there a bug in EngFlow’s remote execution platform?

At EngFlow, that question is treated like a body on the floor. We escalated immediately.

Eliminating the obvious suspects

The initial investigation followed the usual playbook. We examined thread dumps. We inspected scheduler behavior. We looked for backpressure, starvation, or deadlock in our own systems.

Nothing.

Schedulers were idle. Workers were available. No queues were backed up. No locks were contested. From EngFlow’s perspective, the system was… boring.

If this were truly an EngFlow-side issue, it was an extraordinarily well-hidden one.

That mismatch—client claims of failure versus server evidence of normalcy—is where build forensics begins.

When logs fail, follow the wire

To resolve the disagreement between the client and the server, we needed ground truth. That meant looking beneath APIs, beneath libraries, beneath assumptions, down at the network itself.

We asked the customer to capture traffic from the Buck2 client using Wireshark. Crucially, this required decrypting TLS traffic by enabling session key logging via the Rust TLS stack (using SSLKEYLOGFILE support). With those keys loaded, Wireshark could finally show us the actual HTTP/2 conversations, not just encrypted noise.

Because Bazel-style remote execution relies heavily on Protocol Buffers, we also had to teach Wireshark just enough Protocol Buffers syntax to decode the relevant messages. Too much, and decoding would fail. Too little, and the picture would remain blurry.

Forensic work is rarely elegant.

The pattern that didn’t fit

HTTP/2 is stateful. Every request and response is part of a stream identified by a stream ID. If you want to understand a hang, you don’t look at individual packets—you reconstruct conversations.

We quickly outgrew Wireshark’s GUI and switched to tshark, extracting and reassembling request/response pairs while accounting for out-of-order delivery.

That’s when the anomaly emerged.

For successful requests, everything looked correct. Requests ended cleanly. Responses flowed. Streams closed.

For hanging requests, something subtle, but fatal, was missing.

The END_STREAM flag.

The missing END_STREAM

In HTTP/2, every request must eventually signal that it has finished sending data. That signal is the END_STREAM flag. Without it, the protocol enters a polite standoff:

  • The client waits for a response.
  • The server waits for the request to finish.

Neither side is wrong. Both were waiting correctly.

We had found the deadlock.

Critically, every EngFlow response was sent as soon as that flag appeared. When it didn’t, EngFlow waited, as it should. The evidence was unambiguous: the server wasn’t stuck. The request was incomplete.

The culprit was not EngFlow.

The real suspect

Tracing the behavior upstream led us to an unexpected place: Rust’s h2 HTTP/2 library, used by the Buck2 client.

Under certain conditions, the client failed to emit the END_STREAM flag. Five percent of the time, the protocol state machine simply… never finished the request.

One of our engineers, Benjamin Peterson, had already identified a potential fix. The patch was accepted upstream into the Rust h2 library, and Meta has since incorporated the fix into Buck2.

Case closed.

Why this story matters

At first glance, this might look like an interesting debugging anecdote. But for senior technical leaders, it illustrates something deeper.

Modern build systems are distributed systems. They run continuously. They depend on networking stacks, protocol implementations, and libraries far below the level most teams think about day-to-day. When something goes wrong, the failure modes are silent, partial, and counterintuitive.

This is why Build Forensics matters.

Build Forensics is not about logs and dashboards alone. It is about:

  • correlating behavior across client and server,
  • understanding protocols at the wire level,
  • reconstructing causality across organizational boundaries,
  • and being willing to follow the evidence—even when it leads outside your codebase.

In this case, solving the problem required:

  • deep knowledge of HTTP/2 semantics,
  • familiarity with Rust’s networking stack,
  • careful TLS decryption and packet reconstruction,
  • and close collaboration across companies.

That is the reality of Build Management at Scale.

EngFlow exists because build infrastructure has crossed a line. It is no longer “tooling.” It is production infrastructure with production-grade failure modes.

Our job isn’t just to keep builds fast. It’s to make them understandable when they fail—and to do the forensic work necessary to restore confidence when the system behaves unexpectedly.

Sometimes the bug is ours. Sometimes it isn’t. Either way, the system doesn’t care.

The only thing that matters is whether someone knows how to solve the case.

If stories like this resonate, you probably already know: build systems are where some of the hardest infrastructure problems now live. If you want to work on them, we’re hiring.