Secure Builds with Credential Helpers¶
Secure and reproducible builds are something we'd all like, and something many of us work on regularly.
One area that is frequently overlooked in the topic is how to securely provide credentials for external services used during a build. Systems often fall back to insecure methods like passing them on the command-line or storing them in plaintext on disk.
In this post, we will provide an overview of common authentication mechanisms and why they should not be considered secure, and then introduce Credential Helpers and how they do provide secure builds.
Historically, compilers and build systems had little to no need to know about the world outside of the local machine, let alone interact with it. With modern build tools like Bazel or Buck2, interchange between the build and other machines became increasingly popular. Common examples for connecting to the internet are for downloading third-party dependencies (e.g, archives from GitHub), using a Remote Cache or even executing builds remotely. In many cases, interacting with such systems requires authentication, so tools must provide a way for users to supply credentials. And since one such mechanism rarely fits everyone's needs, many tools started integrating more and more ways for their users to specify credentials.
Status Quo¶
For Bazel, this journey started with support for Google Cloud Platform. Later, Bazel grew support for .netrc files, mTLS, specifying flags on the command line with --remote_header and --bes_header, and many more requested over the years.
While these ways to specify credentials provide a sufficient way for most users
to inject credentials, they are insufficient to satisfy everyone's needs, and
not all mechanisms apply to all ways of interacting with remote services
(e.g., did you know that .netrc credentials did not apply to --bes_backend
before 1c14552), and using some combinations is just plain
impossible.
With all these limitations and caveats, a common recommendation is to pass
--{bes,remote}_header to Bazel on the command-line or from .bazelrc files.
While this solves authentication, it introduces a new issue: leaking your
credentials! This is already a big issue for CI where the command-line is
typically shown in logs, but its severity has further increased with the rise of
platforms leveraging the Build Event Protocol to show build and test
results in a user-friendly way: these platforms often show the full command-line
of the build.
While most commercial service providers do take the risk of leaking credentials
into account, lack of standardization makes it impossible to use heuristics to
redact all values for --{bes,remote}_header. Not to mention
--remote_downloader_header, --remote_exec_header and
--remote_cache_header, which can also contain secrets.
Credential Helper to the Rescue¶
With such a variety of authentication mechanisms, it is increasingly infeasible for build systems to support the authentication needs of their users.
To prevent this situation of maximizing the number of ways a build system can interact with a remote service while also keeping maintenance cost under control, many tools use a Credential Helper, a binary that acts as secure proxy between the tool and the authentication provider, which allows tools to gather credentials using a simple and vendor-independent protocol.
However, while tools like git or docker successfully adopted Credential Helpers a long time ago, and many vendors provide a Credential Helper for their service to their customers, build systems have historically lacked such capabilities and resorted to a limited number of built-in authentication mechanisms.
So Yannic, wouldn't it be great if Bazel supported the same, rich ecosystem?
yes! Of course!, so that's why we created and implemented the Bazel Credential Helper, which is used by many developers every day.
But Yannic, what about the larger ecosystem of build systems beyond Bazel? Can't they benefit from the same set of Credential Helpers that exist for Bazel? Many of them already use the same protocol to interact with remote services like execution or caching.
Absolutely! Today we are happy to announce our plans to standardize the specification and work with maintainers of build systems and other tools to improve and extend the specification.
Consequently, we've compiled the original proposal for Credential Helpers in Bazel into a formal specification, which can be found at github.com/EngFlow/credential-helper-spec.
We are also happy to announce our plans to bring Credential Helpers to all our favorite build systems, starting with reclient and Buck2. Please reach out if you are interested in support for Credential Helpers in other build systems or tools.
Using Credential Helpers¶
Now that we have learned about what Credential Helpers are and why they provide a better experience to developers and are more secure than their alternatives, let's dive into how to use them. As a prerequisite, you'll need a Credential Helper binary which is typically distributed by the vendor of the remote service. If the vendor does not provide a Credential Helper, we provide instructions for creating one in the section below.
Once you have the Credential Helper for the remote service, all that's left is to configure the build system. For Bazel, you can, and should, start using Credential Helpers today. We've compiled a short introduction and instructions in a recent blog post. For other build systems, starting with Buck2 and reclient, we are working on getting support for Credential Helpers implemented upstream and into a release.
Creating Credential Helpers¶
Warning
Most people do not need to create their own Credential Helper. Please contact the vendor of the remote service before creating your own.
For folks needing to create their own Credential Helper, the specification provides a comprehensive set of instructions.
However, we understand that reading the specification may seem daunting, so we created the Go and Rust SDK to simplify development of new Credential Helpers and accelerate adoption, licensed under Apache 2.0 and ready for use.
We are also working on SDKs for these other languages, which we expect to publish soon.
- C++
- Node.JS