Reduce LLVM Build Artifact Storage Costs by 50% with Content-Defined Chunking
One of the classic strategies to speed up a system is to avoid redundant work. Scalable build systems like Bazel and Buck2 heavily employ this strategy in various ways, with remote caching and content-addressable storage being two prominent examples. While remote caching prevents repeating redundant build actions, content-addressable storage (CAS) exists for the purpose of data deduplication. However, traditionally CAS operates at the granularity of a single file. When you modify a single byte in a file and store it in a CAS, the CAS stores a second, complete file. Deduplication at the file level is quite palatable for smaller files; a single build invocation typically contains many thousands of small files. Who cares if we store a couple more?
However, as file sizes increase, the cost of storing yet another slightly modified version of a file becomes more expensive. Instead of tossing a couple extra kilobytes into storage, you might be storing a few more gigabytes. Now, consider where these large files come from. These large files are often outputs of build actions, and those build actions depend on many smaller inputs. As those many inputs churn, the outputs also churn. Suddenly the cost of touching a tiny little source file isn’t just the cost of uploading a new version of the source file to CAS–it’s now also the cost of all the large outputs that are produced by the build. This dictates the growth rate of your CAS storage costs, which scales directly with the number of incoming builds. At AI-scale, these costs have become more important than ever before.
Please hold off on pushing your PRs for a bit until the patch release is fully cut. We don't have enough capacity to handle all the load and the patch is blocked because of that.