Operations¶
Components¶
A deployment of TRS-Filer consists of:
- the app (the TRS API), served by Gunicorn (see Server);
- MongoDB, which stores the tools, with their versions and files, and the
tool classes (database
trsStore, collectionstoolsandtoolclasses, andpolicy_ruleswith the rules of access control; see Database); - an OpenID Connect provider of the deployment, whose access tokens TRS-Filer validates for writes (see Access control);
- optionally, a bucket of S3-compatible object storage, which then stores the content of files (see File storage).
The Docker Compose setup and the Helm chart run the app and MongoDB; neither bundles an object store.
Logs¶
The app logs to standard error (section log of the config, by default at
level INFO): each request to the API (method, route and client address,
not the body; the path at DEBUG) and the status of its response (except
for errors, see below), registered and replaced objects, and errors, e.g.,
the reasons why the content of a file could not be retrieved from its URL
(which clients don't get, see Remote files).
Read them with docker compose logs or kubectl logs.
Requests that fail with an error have no line with the status of their
response. Requests that don't match the API specification have no line of
the request either: they are logged at INFO as Request rejected:, with
the locations and codes of the invalid fields where known (otherwise a
general reason, e.g., The request is malformed.), and the reason of the
validation only at DEBUG.
Other errors that FOCA's error handler answers (e.g., 404 Not Found,
500 Internal Server Error, and 400 Bad Request for files whose content
can't be retrieved) are logged at ERROR with the problem details sent,
which include the status; unknown routes have no line of the request.
TRS-Filer's own 400 Bad Request responses to requests it refuses (see
Errors) are logged only with the reason, at ERROR (e.g.,
Invalid limit 0.), after the line of the request.
Health checks¶
The image checks its health by requesting the Swagger UI
(/ga4gh/trs/v2/ui/), which needs neither the database nor authentication;
set HEALTHCHECK_URL if Gunicorn's port, the API base path or the Swagger
UI setting changes. On startup, each Gunicorn worker creates the app and
checks the config (e.g., the request size limit and the custom config);
if this fails, Gunicorn exits.
Backups¶
All state is in MongoDB, except the content of files in object storage (see
below): back up the database (trsStore, or the one set in
MONGO_DBNAME; trsStore with the Helm chart) with
mongodump, e.g.,
in the Docker Compose setup:
docker compose exec -T mongodb mongodump --db trsStore --archive > trs-filer.archive
docker compose exec -T mongodb mongorestore --archive < trs-filer.archive
mongodump reads the collections one after the other, without a
point-in-time snapshot, so a registration during the dump can leave the
archive inconsistent: stop writes (e.g., stop the app with
docker compose stop trs-filer) while dumping, or use a backup method of
MongoDB that takes consistent snapshots.
The base config creates unique indexes on the identifiers (id) of tools
and tool classes on startup, sparse indexes on their pending flags
(_pending, see Adding tool classes),
and a sparse index on the keys of the objects
of files in object storage (versions.files._storage.key of tools, see
File storage), which the check of released
objects on every write uses.
With object storage (custom.storage.backend: s3, see File
storage), back up the bucket (the objects
with the prefix of custom.storage.s3.prefix) together with the database,
with the tools of the object storage (e.g., versioning or replication of
the bucket, or aws s3 sync). Objects are written once and never changed,
so back up the database first, then the bucket: objects of files
registered meanwhile are then in the bucket's backup, too. Objects of
versions replaced or deleted meanwhile are deleted, though, so enable
versioning (or object retention) of the bucket, to restore the objects
that the database's backup refers to. Restored files
whose object is missing fail with 500 Internal Server Error. Objects that
no file refers to (e.g., left by a crash during a registration) can be
found by comparing the keys in the bucket with those in the database
(versions.files._storage.key of collection tools), and deleted.
Permissions API¶
FOCA also serves its permissions API at /admin/access-control/permissions
(not part of the TRS API specification), which lists, adds, changes and
deletes the stored policy rules directly. Only admins can use it. The
configured policies (security.access_control.policies) are managed on
startup and can't be changed via the API; rules added via the API (e.g.,
{"policy_type": "g", "rule": {"v0": "user:<user-id>", "v1": "admin"}},
which makes a user an admin) apply until they are deleted, also after the
group mapping changes, so prefer groups (see Admins).
The API is served like the TRS API (e.g., the Helm chart's Ingress routes
all paths); to keep it off the internet, route only /ga4gh/trs/v2/
through a reverse proxy.
Verifying images and charts¶
From the first release on, images on Docker Hub (elixircloud/trs-filer)
are built by this repository's workflows on GitHub Actions, with an SBOM and
the build provenance attached as attestations, a GitHub artifact
attestation
of their provenance, and a keyless Sigstore
signature. To check that an image was built by the release workflow, run
(with the GitHub CLI or
cosign 3 or
later, which reads the signatures in the Sigstore bundle format):
gh attestation verify oci://docker.io/elixircloud/trs-filer:0.5.0 \
--repo elixir-cloud-aai/trs-filer \
--signer-workflow elixir-cloud-aai/trs-filer/.github/workflows/release.yml \
--source-ref refs/tags/v0.5.0
cosign verify docker.io/elixircloud/trs-filer:0.5.0 \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity https://github.com/elixir-cloud-aai/trs-filer/.github/workflows/release.yml@refs/tags/v0.5.0
These are the commands for the latest release, 0.5.0: replace it
with the version to verify. --source-ref and the ref in
--certificate-identity check that the workflow ran for the release tag,
not for another branch. Images of commits on dev (dev, sha-<commit>)
are signed and attested by the CI workflow instead: use --signer-workflow
elixir-cloud-aai/trs-filer/.github/workflows/ci.yml,
--source-ref refs/heads/dev and --certificate-identity
https://github.com/elixir-cloud-aai/trs-filer/.github/workflows/ci.yml@refs/heads/dev.
Images published before the workflows signed them (the dated tags and the
latest image of earlier versions, and dev and sha-<commit> images
built before signing was added, see
#134) have no
signature or attestation. To show an image's SBOM or
provenance, run:
docker buildx imagetools inspect docker.io/elixircloud/trs-filer:0.5.0 \
--format '{{ json .SBOM }}'
docker buildx imagetools inspect docker.io/elixircloud/trs-filer:0.5.0 \
--format '{{ json .Provenance }}'
The Helm chart attached to a release (trs-filer-0.5.0.tgz)
has a GitHub artifact attestation of its provenance and a Sigstore bundle
with its signature (trs-filer-0.5.0.tgz.sigstore.json,
attached too):
gh release download v0.5.0 --repo elixir-cloud-aai/trs-filer \
--pattern 'trs-filer-0.5.0.tgz*'
gh attestation verify trs-filer-0.5.0.tgz \
--repo elixir-cloud-aai/trs-filer \
--signer-workflow elixir-cloud-aai/trs-filer/.github/workflows/release.yml \
--source-ref refs/tags/v0.5.0
cosign verify-blob trs-filer-0.5.0.tgz \
--bundle trs-filer-0.5.0.tgz.sigstore.json \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity https://github.com/elixir-cloud-aai/trs-filer/.github/workflows/release.yml@refs/tags/v0.5.0
Images are scanned for known vulnerabilities with Trivy
in CI and again before a release is published (both fail on fixable
critical and high vulnerabilities), and the latest image weekly. The
repository's code scanning alerts show the findings that have a fix, which
the maintainers can act on; findings without a fix (often OS packages of
the base image) appear there once a fix is released. The weekly scan also
keeps a full report, those findings included, as a workflow artifact of
each run. The published images (releases and dev) are the scanned ones:
the workflows push each image by digest, untagged, scan exactly that
digest, and only if the scan passes (no fixable critical or high
vulnerabilities, except accepted ones) sign and tag it (see Versioning and
releases).
Images published to another Docker Hub organization (forks that set
DOCKERHUB_ORG) are verified the same way, with that organization and the
fork's repository in place of elixircloud and elixir-cloud-aai/trs-filer.