Server configuration

BuildGrid’s server YAML configuration format details.

Hint

In order to spin-up a server instance using a given server.conf configuration file, run:

bgd server start server.conf

Please refer to the CLI reference section for command line interface details.

Reference configuration

Below is an example of the full configuration reference:

##
# Server's configuration desciption.
description: |
  BuildGrid's server reference configuration.

##
# Server's network configuration.
server:
  - !channel
    ##
    # TCP port number.
    port: 50051
    ##
    # Whether or not to activate SSL/TLS encryption.
    insecure-mode: true
    ##
    # SSL/TLS credentials.
    credentials:
      tls-server-key: !expand-path ~/.config/buildgrid/server.key
      tls-server-cert: !expand-path ~/.config/buildgrid/server.cert
      tls-client-certs: !expand-path ~/.config/buildgrid/client.cert

##
# Server's authorization configuration.
authorization:
  ##
  # Type of authorization method.
  #  none  - Bypass the authorization mechanism
  #  jwt   - OAuth 2.0 bearer with JWT tokens
  method: jwt
  ##
  # Location for the file containing the secret, pass
  # or key needed by 'method' to authorize requests.
  secret: !expand-path ~/.config/buildgrid/auth.secret
  ##
  # Encryption algorithm to be used together with 'secret'
  # by 'method' to authorize requests (optinal).
  #  hs256  - HMAC+SHA-256 for JWT method
  #  rs256  - RSASSA-PKCS1-v1_5+SHA-256 for JWT method
  algorithm: rs256

##
# Server's instances configuration.
instances:
  - name: main
    description: |
      The 'main' server instance.
    ##
    # List of storage backends for the instance.
    #  disk         - On-disk storage.
    #  lru-storage  - In-memory storage (non-persistant).
    #  remote       - Proxy to remote storage.
    #  s3-storage   - Amazon S3 storage.
    storages:
      - !disk-storage &main-storage
        ##
        # Path to the local storage folder.
        path: !expand-path $HOME/cas
    ##
    # List of services for the instance.
    #  action-cache     - REAPI ActionCache service.
    #  bytestream       - Google APIs ByteStream service.
    #  cas              - REAPI ContentAddressableStorage service.
    #  execution        - REAPI Execution + RWAPI ots services.
    #  reference-cache  - BuildStream ReferenceStorage service.
    services:
      - !action-cache &main-action
        ##
        # Alias to a storage backend, see 'storages'.
        storage: *memory-storage
        ##
        # Maximum number of entires kept in cache.
        max-cached-refs: 256
        ##
        # Whether or not writing to the cache is allowed.
        allow-updates: true
        ##
        # Whether failed actions (non-zero exit code) are stored.
        cache-failed-actions: true

      - !execution
        ##
        # Alias to a storage backend, see 'storages'.
        storage: *main-storage
        ##
        # Alias to an action-cache service.
        action-cache: *main-action
        ##
        # Base URL for external build action (web) browser service.
        action-browser-url: http://localhost:8080

      - !cas
        ##
        # Alias to a storage backend, see 'storages'.
        storage: *main-storage

      - !bytestream
        ##
        # Alias to a storage backend, see 'storages'.
        storage: *main-storage

      - !reference-cache
        ##
        # Alias to a storage backend, see 'storages'.
        storage: *main-storage
        ##
        # Maximum number of entires kept in cache.
        max-cached-refs: 256
        ##
        # Whether or not writing to the cache is allowed.
        allow-updates: true

##
# Server's internal monitoring configuration.
monitoring:
  ##
  # Whether or not to activate the monitoring subsytem.
  enabled: false

  ##
  # Type of the monitoring bus endpoint.
  #  stdout  - Standard output stream.
  #  file    - On-disk file.
  #  socket  - UNIX domain socket.
  endpoint-type: socket

  ##
  # Location for the monitoring bus endpoint. Only
  # necessary for 'file' and 'socket' `endpoint-type`.
  # Full path is expected for 'file', name
  # only for 'socket'.
  endpoint-location: monitoring_bus_socket

  ##
  # Messages serialisation format.
  #  binary  - Protobuf binary format.
  #  json    - JSON format.
  #  statsd  - StatsD format. Only metrics are kept - logs are dropped.
  serialization-format: binary

##
# Maximum number of gRPC threads. Defaults to 5 times
# the CPU count if not specifed. A minimum of 5 is
# enforced, whatever the configuration is.
thread-pool-size: 20

Parser API

The tagged YAML nodes in the reference above are handled by the YAML parser using the following set of objects:

class buildgrid._app.settings.parser.YamlFactory

Bases: yaml.YAMLObject

Base class for contructing maps or scalars from tags.

classmethod from_yaml(loader, node)

Convert a representation node to a Python object.

class buildgrid._app.settings.parser.Channel(port, insecure_mode, credentials=None)

Bases: buildgrid._app.settings.parser.YamlFactory

Creates a GRPC channel.

The Channel class returns a grpc.Channel and is generated from the tag !channel. Creates either a secure or insecure channel.

Parameters
  • port (int) – A port for the channel.

  • insecure_mode (bool) – If True, generates an insecure channel, even if there are

credentials. Defaults to True.

credentials (dict, optional): A dictionary in the form:

tls-server-key: /path/to/server-key
tls-server-cert: /path/to/server-cert
tls-client-certs: /path/to/client-certs
yaml_tag = '!channel'
class buildgrid._app.settings.parser.ExpandPath

Bases: buildgrid._app.settings.parser.YamlFactory

Returns a string of the user’s path after expansion.

The ExpandPath class returns a string and is generated from the tag !expand-path.

Parameters

path (str) – Can be used with strings such as: ~/dir/to/something or $HOME/certs

yaml_tag = '!expand-path'
class buildgrid._app.settings.parser.ReadFile

Bases: buildgrid._app.settings.parser.YamlFactory

Returns a string of the contents of the specified file.

The ReadFile class returns a string and is generated from the tag !read-file.

Parameters

path (str) – Can be used with strings such as: ~/path/to/some/file or $HOME/myfile or /path/to/file

yaml_tag = '!read-file'
class buildgrid._app.settings.parser.Disk

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.disk.DiskStorage using the tag !disk-storage.

Parameters

path (str) – Path to directory to storage.

yaml_tag = '!disk-storage'
class buildgrid._app.settings.parser.LRU

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.lru_memory_cache.LRUMemoryCache using the tag !lru-storage.

Parameters

size (int) – Size e.g 10kb. Size parsed with buildgrid._app.settings.parser._parse_size().

yaml_tag = '!lru-storage'
class buildgrid._app.settings.parser.S3

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.s3.S3Storage using the tag !s3-storage.

Parameters
  • bucket (str) – Name of bucket

  • endpoint (str) – URL of endpoint.

  • access-key (str) – S3-ACCESS-KEY

  • secret-key (str) – S3-SECRET-KEY

yaml_tag = '!s3-storage'
class buildgrid._app.settings.parser.Remote

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.remote.RemoteStorage using the tag !remote-storage.

Parameters
  • url (str) – URL to remote storage. If used with https, needs credentials.

  • instance_name (str) – Instance of the remote to connect to.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-client-key: /path/to/client-key
    tls-client-cert: /path/to/client-cert
    tls-server-cert: /path/to/server-cert
    

yaml_tag = '!remote-storage'
class buildgrid._app.settings.parser.WithCache

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.storage.with_cache.WithCacheStorage using the tag !with-cache-storage.

Parameters
  • url (str) – URL to remote storage. If used with https, needs credentials.

  • instance_name (str) – Instance of the remote to connect to.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-client-key: /path/to/certs
    tls-client-cert: /path/to/certs
    tls-server-cert: /path/to/certs
    

yaml_tag = '!with-cache-storage'
class buildgrid._app.settings.parser.Execution

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.execution.service.ExecutionService using the tag !execution.

Parameters
yaml_tag = '!execution'
class buildgrid._app.settings.parser.Action

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.service.ActionCacheService using the tag !action-cache.

Parameters
  • storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use.

  • max_cached_refs (int) – Max number of cached actions.

  • allow_updates (bool) – Allow updates pushed to CAS. Defaults to True.

  • cache_failed_actions (bool) – Whether to store failed (non-zero exit code) actions. Default to True.

yaml_tag = '!action-cache'
class buildgrid._app.settings.parser.RemoteAction

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.actioncache.remote.RemoteActionCache using the tag !remote-action-cache.

Parameters
  • url (str) – URL to remote action cache. If used with https, needs credentials.

  • instance_name (str) – Instance of the remote to connect to.

  • credentials (dict, optional) –

    A dictionary in the form:

    tls-client-key: /path/to/client-key
    tls-client-cert: /path/to/client-cert
    tls-server-cert: /path/to/server-cert
    

yaml_tag = '!remote-action-cache'
class buildgrid._app.settings.parser.Reference

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.referencestorage.service.ReferenceStorageService using the tag !reference-cache.

Parameters
yaml_tag = '!reference-cache'
class buildgrid._app.settings.parser.CAS

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.service.ContentAddressableStorageService using the tag !cas.

Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use.

yaml_tag = '!cas'
class buildgrid._app.settings.parser.ByteStream

Bases: buildgrid._app.settings.parser.YamlFactory

Generates buildgrid.server.cas.service.ByteStreamService using the tag !bytestream.

Parameters

storage (buildgrid.server.cas.storage.storage_abc.StorageABC) – Instance of storage to use.

yaml_tag = '!bytestream'
buildgrid._app.settings.parser.get_parser()