How to: Build Chromium with Reclient¶

It is official: Goma is no longer supported by Google, and all Chromium and AOSP builds must use Reclient for remote execution and caching.

Reclient is a build system plugin providing remote execution and remote caching capabilities to existing build systems that do not natively support them -- such as Ninja, the build system used to build Chromium. See my last post, Goma is Gone — Put Everything Into Reclient! for an overview of Reclient architecture and some insights into the migration.

In this post we delve into the technical details of how to build Chromium with Reclient on a remote cluster with Linux workers.

Configuring Reclient¶

Reclient consists of a few processes:

reproxy: A local gRPC server handling command execution requests from rewrapper. It is started at the beginning of the build and shut down at the end of the build.
rewrapper: A process that "wraps" every compiler invocation in a request to reproxy, in order to execute the command remotely and/or fetch results from the Action Cache.

How Reclient reads options¶

All Reclient processes accept options in three ways:

Command line, like reproxy --service=your.remote.cluster.address:port
Environment variables starting with the RBE_ prefix, like export RBE_service=your.remote.cluster.address:port.
Configuration file, like reproxy --cfg=/some/path/to/reproxy.cfg, where the file contains options as:
```
# Comments are allowed
service=your.remote.cluster.address:port
```

We strongly advise you to only set each option in one place. If a option is set in more than one place, the list above is ordered by precedence: command line options override environment variables, and environment variables override configuration files.

Note

All Reclient processes write their finalized parsed options in their logs on startup.

Authentication¶

Reclient supports multiple methods of authenticating to the remote server (and we have work under way to add Bazel Credential Helper support as well). Currently, we recommend our customers use mTLS for authentication by setting the relevant options via environment variables:

Bash

export RBE_service=your.remote.cluster.address:port    # change this to your endpoint
export RBE_tls_client_auth_cert=/path/to/engflow.crt   # change these to your path
export RBE_tls_client_auth_key=/path/to/engflow.key
# These options should not be required, but are required for now:
export RBE_service_no_auth=true
export RBE_use_application_default_credentials=true

Chromium: Reclient configuration files¶

In the Chromium build, all Reclient binaries are invoked with the --cfg option pointing to a fixed path in the build tree. For example, reproxy expects to find a configuration file at buildtools/reclient_cfgs/reproxy.cfg.

In general, Chromium build requires the following files:

One configuration file for reproxy
One configuration file for rewrapper per combination of tool and target OS (for example, tool can be python and target OS can be linux).

Chromium generates these files by running a script as part of DEPS to download the latest version of the tooling and the final configuration files (which then happens on every gclient sync).

Engflow: Reclient configuration script¶

We have built the open source reclient-configs repository to make it easy to extend and modify the Google configuration files for Reclient. You can add a reference to our repository from the DEPS file like so:

DEPS: deps variable

'src/third_party/reclient_configs': 'https://github.com/EngFlow/reclient-configs.git',

We recommend pinning the repository to a particular commit for consistency by adding a @<commit-hash> suffix.

Then, append the following stanza to the end of the hooks variable in the DEPS file, to be run on gclient sync:

DEPS: last hook

  {
    'name': 'configure_reclient',
    'pattern': '.',
    'action': ['python3', 'src/third_party/reclient_configs/configure_reclient.py', '--src_dir=src'],
  },

Now, every time you run gclient sync, you should see the new configuration files generated under buildtools/reclient_cfgs:

buildtools/reclient_cfgs/reproxy.cfg
buildtools/reclient_cfgs/chromium-browser-clang/
    rewrapper_linux.cfg
    rewrapper_mac.cfg
    rewrapper_windows.cfg
buildtools/reclient_cfgs/nacl/
    rewrapper_linux.cfg
    rewrapper_mac.cfg
    rewrapper_windows.cfg
buildtools/reclient_cfgs/python/
    rewrapper_linux.cfg
    rewrapper_mac.cfg
    rewrapper_windows.cfg

See reclient-configs/README for details on how to easily customize the generated configuration files, if needed for your particular use-case -- for instance, see Brave's example for a relatively complex use-case.

Sanity check: verify that Reclient works¶

We have a handy check_reclient_works.py script to verify that a single remote action works end-to-end with Reclient. You can invoke it like so:

Bash

python3 third_party/reclient-configs/check_reclient_works.py --src_dir=.

Running the Chromium build¶

Run gn:

Bash

gn args out/reclient

Use use_remoteexec = true in the args to enable Reclient builds. We recommend starting with a small target at first:

Bash

autoninja -C out/reclient base

Run the full build with:

Bash

autoninja -C out/reclient chrome

Build status: what is happening¶

To monitor a running build, you can use the buildtools/reclient/reproxystatus binary:

Bash

watch buildtools/reclient/reproxystatus

We also have work in progress for adding BEP support to Reclient, which will enable us to provide a UI for Reclient builds, similarly to what we have for Bazel builds -- see my talk from the recent Build Meetup for a teaser!

Debugging - Reclient logs¶

The Reclient processes output all logs to the .reproxy_tmp/logs directory under the build output directory, like out/Default/.reproxy_tmp/logs. The verbosity can be controlled with --v and --vmodule options. Specifically, we find the following logs very useful:

reproxy_<timestamp>.rrpl (structured log of every single action processed by Reclient).
reproxy.INFO (text log of the reproxy process).
rbe_metrics.txt (generated at the end of the build, aggregation of reproxy_<timestamp>.rrpl).

What's next¶

In followup blog posts, we plan to do more deep dives on:

Reclient features / FAQ
Reclient performance tuning

Contact us if you have more Reclient questions / topics you'd like to see!