BuildStream client

BuildStream is a free software tool for building and integrating software stacks. It supports remote build execution using the remote execution API (REAPI) v2. The project’s documentation has a detailed section about its remote execution subsystem architecture that you are very recommanded to read first.

Note

There is no stable release of BuildStream with support for remote execution yet. You’ll have to install it from sources in order to build with remote execution.

Configuration

BuildStream uses YAML for build definition and configuration. It has two levels of configuration: user and project level. User-level configuration is stored in your buildstream.conf file while project-level configuration is defined in each project’s project.conf file.

Note

At the moment, remote execution can only be configured at project-level.

Project configuration

In order to activate remote build execution at project-level, the project’s project.conf file must declare two specific configuration nodes:

Important

BuildStream does not support multi-instance remote execution servers and will always submit remote execution request omitting the instance name parameter. Thus, you must declare an unnamed ‘’ instance in your server configuration to workaround this.

Important

If you are using BuildStream’s artifact server, the server instance pointed by the storage-service key must accept pushes from your client for remote execution to be possible.

BuildBox tool

BuildStream performs builds in a sandbox on top of a project-defined environment, not relying on any host-tools. BuildGrid supports this kind of build using the buildbox bot, a specific type of bot relying on BuildBox for build execution.

BuildBox can execute build commands in a sandbox on top of an environment constructed from provided sources using FUSE. It also uses bubblewrap for sandboxing without requiring root privileges.

BuildBox being a rather young project, it isn’t packaged yet and you’ll have to build it from source. You may want follow the manual instructions or you can build it with BuildStream using the dedicated integration project (recommanded).

Important

Whatever the method you use to install BuildBox, you also have to install bubblewrap along, minimum required version being 0.1.8.

Example build

The BuildStream repository contains example projects used for testing purpose in the project’s usage documentation section. We’ll focus here on instructions on how to build the autotools example running BuildStream and BuildGrid on your local machine, compiling the GNU Automake hello example program in a sandbox on top of a minimal Alpine Linux environment.

First, you need to checkout the buildstream repository sources:

git clone https://gitlab.com/BuildStream/buildstream.git

Next, change the current directory to the BuildStream project root:

cd buildstream/doc/examples/autotools

Hint

All the commands in the instructions below are expected to be executed from that root directory (the autotools example project’s root directory).

Before running BuildStream and building the example project, you’ll have to setup and run a BuildGrid server and bot. A minimal server’s configuration is given below, paste it in a server.conf file in the root directory:

server:
  - !channel
    port: 50051
    insecure-mode: true

instances:
  - name: ''

    storages:
      - !lru-storage &main-storage
        size: 512MB

    services:
      - !action-cache &main-action
        storage: *main-storage
        max-cached-refs: 256
        allow-updates: true

      - !execution
        storage: *main-storage
        action-cache: *main-action

      - !cas
        storage: *main-storage

      - !bytestream
        storage: *main-storage

      - !reference-cache
        storage: *main-storage
        max-cached-refs: 128

This defines a single unnamed server instance implementing a ContentAddressableStorage (CAS) + Reference + ByteStream service together with an Execution + ActionCache service, both using the same in-memory storage. You can then start the BuildGrid server daemon using that configuration by running:

bgd server start server.conf

In order to perform the actual build work, you need to attach a buildbox worker bot to that server for that unnamed instance. Once you’ve make sure that the buildbox tool is functional on your machine (refer to configuration), run:

bgd bot --remote=http://localhost:50051 --parent= buildbox --fuse-dir=fuse --local-cas=cache

The --remote option is used to specify the server location (running on the same machine here, and listening to port 50051). The --parent option is used to specify the server instance you expect the bot to be attached to (empty here). --fuse-dir and --local-cas are specific to the buildbox bot and respectively specify the sandbox mount point and local CAS cache locations. Refer to the CLI reference section for for command line interface details.

The BuildGrid server is now ready to accept jobs and execute them. The example project needs some configuration tweaks in order to be build remotely. Below is the configuration fragment you should append at the end of the project.conf file from the root directory:

artifacts:
  url: http://localhost:50051
  push: true

remote-execution:
  execution-service:
    url: http://localhost:50051
  storage-service:
    url: http://localhost:50051
    client-key: ''
    client-cert: ''
    server-cert: ''
  action-cache-service:
    url: http://localhost:50051

This activates BuildGrid’s remote execution mode and points to the unnamed remote execution server instance at localhost:50051.

You can finally have BuildStream to build the example project by running:

bst build hello.bst

You can verify that the example has been successfully built by running the generated executable. Simply invoke:

bst shell hello.bst -- hello