Build Forensics: How we uncovered a Rust HTTP/2 bug¶
About 5 months ago, one of our customers began reporting that 5% of their jobs based on Meta's Buck2 build system were failing. Even though there was no simple way to reproduce the phenomenon, all signs pointed to network hangs when a Buck2 client made API calls to their EngFlow cluster. If a simple timeout was added, the request would quickly time out, potentially indicating the issue was a bug in our remote execution platform.
Because of the potential for it to expose a broader issue in our software, especially in terms of compatibility issues with Buck2, we elevated the urgency of this investigation. After examining a few thread dumps that revealed nothing conclusive, we eventually scheduled a call with the customer. After performing multiple tests, it became evident that our schedulers were simply idle. If it were truly a problem in our system, we were unable to find evidence.
Going Deep: Network Packet Analysis¶
In order to resolve this inherent disagreement between the Buck2 client and our server, we asked the customer to perform network captures on the client side. Wireshark has the ability to not only capture network packets but also decrypt Transport Layer Security (TLS) packets so long as the ephemeral session keys are logged, which can be done by setting the SSLKEYLOGFILE environment variable in some applications or in this specific case, enabling this option in the Rust TLS library. This log file can then be specified in Wireshark to decipher what was happening in the encrypted exchange.
Because much of Bazel's remote execution APIs rely on Google's protocol buffers, Wireshark also needed to be configured to decode them. Wireshark actually doesn't depend on the complete C++ protobuf compiler but instead relies on a simple grammar to decode these files, so we had to specify the minimal set of protocol buffers needed. If we included any protocol buffer that Wireshark couldn't interpret, the loading would fail!
Unraveling the HTTP/2 Mystery¶
HTTP/2 is inherently a stateful protocol, so deciphering what happened required investigating what happened with each request and response pair. Each HTTP/2 request has an associated stream ID, so uncovering the mystery required unraveling what was happening for hanging requests. When Wireshark's GUI-driven interface didn't allow for this in-depth analysis, we switched to using the terminal counterpart tshark to parse out this data. In both cases, we had to make sure that all packets were reassembled in case they arrived out-of-order.
Ultimately, we discovered a glaring issue in the HTTP/2 request and response pairs. Normally at the end of each HTTP/2 request, the END_STREAM flag must be set. In cases where the request was hanging, we noticed that this flag was not being sent.
The Deadlock Revealed¶
In other words, we had essentially identified a deadlock. The Buck2 client was waiting to finish sending the request, but the server was also waiting for the client. Because we were able to show that every response from the EngFlow side arrived whenever this flag was set, it pointed to a problem in the Buck2 client.
The root cause was eventually pinpointed to be a bug in Rust's HTTP/2 library, h2. One of our engineers, Benjamin Peterson, had already issued a potential fix, which was independently identified and confirmed by Brian Vincent with a reproducible test case. The core patch has now been incorporated into the Rust h2 library, and Meta earlier this year incorporated this fix into Buck2.
Building Better Infrastructure Together¶
We hope this blog post showcases some of the unique challenges we face that often depend on the strong collaboration across multiple companies to resolve. We relied on using Wireshark to understand the root cause but we needed our customers' help to test and verify any potential fixes. Ultimately, the problem was exposed in a core Rust library that the Buck2 client uses, which we never would have expected when we first began investigating this issue.
If you're an engineer interested in solving these kinds of infrastructure and systems problems for our customers, please apply! We also hope this gives a sense of the technical issues we face, and how we are not only improving the Bazel ecosystem but any clients that support the Remote Execution APIs such as Buck2. Understanding what happened required deep domain knowledge around the HTTP/2 specification and digging deep into network internals. We also upstreamed a few improvements to Wireshark including these minor fixes (1, 2) along the way.