Goma is Gone — Put Everything Into Reclient!

On September 7th, Google officially announced the deprecation of the Goma remote execution platform used to accelerate Chromium builds. Chromium is migrating to Reclient, a more modern tool compatible with the Remote Execution API (RE-API) used by Bazel, Buck2, recc, and other tools.

We are writing to inform you that we plan to delete support for Goma in Chromium builds around the beginning of 2024. We will also stop maintaining the goma client and server.

If you use Goma to build a Chromium browser, this may seem like bad news — and it's true that it may cause you some disruption in the short term. However in the longer term, the benefits of switching to Reclient vastly outweigh the costs. This transition has been in the works for a long time, and the deprecation notice was only sent out after Reclient has been proven to work smoothly, with minimal client-side migration requirements.

Before joining EngFlow, I worked in Google's RBE team, the team behind the RE-API and Google's internal RE backend. During my last years at Google I was part of the team developing Reclient, and since joining EngFlow, I've supported our Goma and RE-API services. I've seen both platforms in action from both client- and server-side. Below, I will try to offer some insight into this migration: what it means, why did we plan it, and how it will impact Chromium developers.

What is Goma?

Goma users might refer to Goma as their "compiler" or "build system", and that is usually a good enough view for practical purposes. However, it is not technically accurate, and here the nuances matter:

• Goma is not a build system. A build system computes and orchestrates a full action graph of the build. Ninja, make, Bazel or Buck2 are example build systems.
• Goma is not a compiler. It makes calls to to compilers, like clang or gcc, but these toolchains are configurable and separate from Goma itself.

So what does Goma actually do?

In a nutshell, Goma is a plugin that provides remote execution and remote caching capabilities to existing build systems that do not natively support them. In particular, this entails:

Input Processing

The main challenge of remote execution and remote caching is to figure out the full set of inputs (also known as include scanning) required to execute an action. As a simple example, consider this program:

#include <stdio.h>

int main(int argc, char** argv) {
  printf("Hello, World!\n");
}

To compile this code we need main.c itself, but that's not all. We also need the included system header stdio.h, and all the files that are included in that header, and so on. When we compile the program on our local machine, we don't think about any of this — we count on all these files "existing somewhere", and it "just works". However, if we want to use remote caching or remote execution, we need to ensure that:

• All the required files exist on the remote worker (otherwise, our remote execution will fail).
• All of them have the exact same paths and exact same contents as our local machine (otherwise, any differences might produce different results and poison our cache).

Goma is extremely good at efficiently solving this problem for C++ compilations. So good, in fact, that the Reclient tool, developed to replace Goma, still uses the Goma include processor under the hood: "Goma's input processor is 3x faster than clang-scan-deps for a typical compile action in Chrome."

Remote Execution

Goma predates the Remote Execution API (RE-API), and uses its own custom remote execution protocol and backend server implementation. After identifying inputs, the Goma client sends the required files to the Goma server, taking advantage of files uploaded previously, then only executes actions if their results are not already remotely cached.

Goma build architecture

What is Reclient? How is it similar and different from Goma?

Reclient is functionally and structurally extremely similar to Goma — with the following key differences:

• Reclient speaks RE-API natively, meaning:
  • it can work with any RE-API compatible backend (and not just Google's).
  • the custom Goma backend is therefore no longer needed, potentially removing its support and maintenance overhead.
• Reclient is much more generic than Goma, handling not only C++ compiles and links, but many other types of actions (think: Java, Python, etc), with easy extension capabilities to support more types, as needed.
• Reclient can integrate with various build systems, not just Chromium's (Gn+Ninja) — for example, Cmake.
• Reclient exports extensive usage metrics and stats, which can be used by various monitoring tools.

Reclient build architecture

In the diagrams of Goma and Reclient above, same colors indicate processes of similar functionality. For example, Reclient's rewrapper process mimics Goma's gomacc, the reproxy process mimics Goma's compiler proxy, etc.

We get it — Reclient is the new shiny. But why deprecate Goma?

To fully understand that, we need a bit of history. Let's start with the RE-API (and I'm going to put my ex-Google-RBE hat on for this).

Google introduced the RE-API, because we wanted to standardize remote caching and execution. Having the API meant that now anyone could implement their own backend, and any build system could integrate with it. Before the standard existed, any build system that wanted remote execution had to also implement the backend, and then maintain it forever, because nothing could be reused.

So naturally, as soon as the first Google RBE implementation was stable, nobody wanted to maintain Goma servers — this was the entire point. But at that point in time, the RBE team was focused on supporting Bazel builds, and couldn't dedicate engineering resources to replacing the Goma client. And so it was decided to deprecate Goma in stages:

First, a proxy-mode will be added to Goma, meaning:

• the Goma client would not change

• the Goma server would still be required, but instead of running remote actions, it would now act strictly as a middleman (proxy mode), translating all requests between the Goma and the RE-API protocols. This would require running less Goma servers, and make them more lightweight, making Goma easier to support.

Secondly, the team would start working on Reclient, which would render the Goma server an entirely unnecessary middleman — at which point it could be safely deprecated.

Goma proxy mode build architecture

However, the temporary "proxy mode" stage ended up lasting for a long time. As the popular saying goes, "Nothing is more permanent than temporary solutions".

There are many problems with running Goma in proxy mode. Some stem from the differences between the two remote execution protocols, which make the "translation layer" extremely inefficient. The RE-API is built on gRPC, which is built on HTTP/2, and the API really depends on the advantages HTTP/2 provides. It means we can multiplex many requests over the same connection, and we can do long-running streaming uploads and downloads. Goma uses plain HTTP/1, so it needs lots of connections, and files need to be broken into chunks for upload and download, which causes lots of complications.

Other problems stem from the Goma servers themselves being difficult to maintain, especially with respect to monitoring and production stability. To sum up a lot of technical details: Goma in proxy mode is heavy, expensive, and error-prone. And few teams had enough incentive to spend resources to improve it, because "it was going away anyway" (for years).

Putting my EngFlow hat back on: at EngFlow, we have spent a lot of time and engineering resources on maintaining Goma servers in proxy mode, and we are really looking forward to being able to permanently delete them and move forward with Reclient.

Fine, Goma is going away. What does this mean for me?

Well, that all depends on how you use Goma:

• Are you maintaining your own Goma servers?
• Or are you using an RE-API backend, using Goma in the afore-mentioned proxy-mode?

If you're running your own Goma servers

I'm afraid I have some bad news: this is not going to be easy! As I see it, you have two options:

1. Migrate to an RE-API backend.
2. Ignore the deprecation and keep maintaining Goma yourself.

The main problem with ignoring the deprecation is not just having to maintain Goma itself — Goma is OSS, after all, and that's not going to change. The real problem is no longer being able to easily update the Chromium source tree. The Goma support is stitched throughout the various build files and scripts in the tree — if you keep a fixed old version of these, it will become more and more difficult to patch Goma support back in, as the tree diverges.

Hence, I believe that only the first option — migrate to an RE-API backend — is a feasible long term solution. Fortunately, there are many existing RE-API-compatible backend solutions. At EngFlow, we obviously hope that you give ours a try — but there are quite a few options to choose from, both OSS and commercial. You could also write your own — but as someone who worked on two of these systems (Google's RBE and now EngFlow), I would strongly suggest to only consider that option as a last resort. It is not an easy one :-)

However, once you get past that hurdle, the client-side migration should be extremely easy.

If you're using an RE-API backend (e.g EngFlow)

Sunshine and rainbows! Your backend already fully supports Reclient. And both Chromium and Android source trees already include full client-side support of Reclient as well. The entire migration process should be as simple as changing a few configuration files. Both Google and EngFlow are in the process of publishing more Reclient documentation, including detailed usage and migration instructions.

Building Chrome with Reclient

EngFlow will be helping all our customers transition to Reclient. In followup blogs, we plan to do deep dives on:

• Detailed migration instructions to Reclient
• Reclient performance tuning
• Reclient features, e.g. tracing, monitoring, correctness proofs

Help, I have more Reclient questions

Stay tuned to hear more about these topics. In the mean time, contact us if you have more questions.