Skip to content

buck2

Build Forensics: How we uncovered a Rust HTTP/2 bug

Last year, one of our customers began reporting that 5% of their jobs based on Meta's Buck2 build system were failing. Even though there was no simple way to reproduce the phenomenon, all signs pointed to network hangs when a Buck2 client made API calls to their EngFlow cluster. If a simple timeout was added, the request would quickly time out, potentially indicating the issue was a bug in our remote execution platform.

Because of the potential for it to expose a broader issue in our software, especially in terms of compatibility issues with Buck2, we elevated the urgency of this investigation. After examining a few thread dumps that revealed nothing conclusive, we eventually scheduled a call with the customer. After performing multiple tests, it became evident that our schedulers were simply idle. If it were truly a problem in our system, we were unable to find evidence.

Secure Builds with Credential Helpers

Secure and reproducible builds are something we'd all like, and something many of us work on regularly.

One area that is frequently overlooked in the topic is how to securely provide credentials for external services used during a build. Systems often fall back to insecure methods like passing them on the command-line or storing them in plaintext on disk.

In this post, we will provide an overview of common authentication mechanisms and why they should not be considered secure, and then introduce Credential Helpers and how they do provide secure builds.