buildgrid.server.cas.storage.index.sql module

SQLIndex

A SQL index implementation. This can be pointed to either a remote SQL server or a local SQLite database.

class buildgrid.server.cas.storage.index.sql.SQLIndex(sql_provider: SqlProvider, storage: StorageABC, *, window_size: int = 1000, inclause_limit: int = -1, max_inline_blob_size: int = 0, refresh_accesstime_older_than: int = 0, **kwargs: Any)

Bases: IndexABC

start() None
stop() None
has_blob(digest: Digest) bool

Return True if the blob with the given instance/digest exists.

get_blob(digest: Digest) IO[bytes] | None

Get a blob from the index or the backing storage. Optionally fallback and repair index

delete_blob(digest: Digest) None

Delete a blob from the index. Return True if the blob was deleted, or False otherwise.

TODO: This method will be promoted to StorageABC in a future commit.

commit_write(digest: Digest, write_session: IO[bytes]) None

Store the contents for a digest.

The storage object is not responsible for verifying that the data written to the write_session actually matches the digest. The caller must do that.

missing_blobs(digests: List[Digest]) List[Digest]

Return a container containing the blobs not present in CAS.

bulk_update_blobs(digest_blob_pairs: List[Tuple[Digest, bytes]]) List[Status]

Implement the StorageABC’s bulk_update_blobs method.

The StorageABC interface takes in a list of digest/blob pairs and returns a list of results. The list of results MUST be ordered to correspond with the order of the input list.

bulk_read_blobs(digests: List[Digest]) Dict[str, bytes]

Given an iterable container of digests, return a {hash: file-like object} dictionary corresponding to the blobs represented by the input digests.

Each file-like object must be readable and seekable.

least_recent_digests() Iterator[Digest]

Generator to iterate through the digests in LRU order

get_total_size(include_marked: bool = True) int

Return the sum of the size of all blobs within the index.

If include_marked is True, it will also include the size of blobs that have been marked deleted.

mark_n_bytes_as_deleted(n_bytes: int, dry_run: bool = False, include_marked: bool = True, protect_blobs_after: datetime | None = None, renew_windows: bool = True) Tuple[List[Digest], List[Digest]]

Mark a given number of bytes’ worth of index entries as deleted.

The entries are marked as deleted in LRU order until the total size marked as deleted is at least the requested number of bytes. If any entries are already marked as deleted, they are used first when attempting to reach the required number of bytes.

Parameters:
  • n_bytes (int) – The number of bytes of index entries to mark as deleted. When the sum of the size_bytes of index entries meets or exceeds this value, the affected digests will be returned.

  • include_marked (bool) – Consider items already marked for deletion.

Returns:

Two lists of digests that were marked as deleted. The first list

contains items that are inlined, the other list contains larger blobs contained in a separate storage.

Return type:

list

bulk_delete(digests: List[Digest]) List[str]

Delete a list of blobs from storage.