Configuration¶
The app is configured by
trs_filer/config.yaml,
the base config, which documents its values. It is a
FOCA config: the sections server, db,
api, service_info, log and exceptions are described in the FOCA
documentation, the app's own
settings are in section custom (validated on startup by
trs_filer.custom_config.CustomConfig).
To override some values, write them to a config overlay file and list it in
environment variable FOCA_CONFIG_OVERLAY (several files separated by :);
FOCA merges overlays into the config on startup (see the FOCA deployment
guide). Mappings are
merged recursively; all other values, including lists, replace those of the
base config.
Deployment configuration¶
TRS-Filer works with any OpenID Connect provider that issues access tokens
as per RFC 9068 (JWTs). The
identity provider, its claims and the groups mapped to roles are deployment
configuration: the base config has no defaults for them, so each deployment
sets them in an overlay (or in the Helm chart's config value),
otherwise the app refuses to start:
security.auth.issuers: the trusted token issuers (theissclaim of their tokens, e.g.,https://login.example.org/oidc/); tokens of other issuers are rejected (see Authentication);security.auth.audience: the client IDs of the apps whose tokens are accepted (theaudclaim, e.g., of a portal or gateway that registers tools), unlesssecurity.auth.verify_audienceisfalse, which accepts tokens for any audience (not recommended);security.access_control.claim_groups: the claim with the groups of users (e.g.,groups), needed for admins; add it tosecurity.auth.userinfo_claimsif the identity provider releases it via user info only, not in access tokens;security.access_control.group_mapping: the group(s) whose members are admins (values of the group claim, see Access control), orsecurity.access_control.require_admin: falseto run without admins (nobody can write then).
For example:
security:
auth:
issuers:
- https://login.example.org/oidc/
audience:
- trs-portal
userinfo_claims:
- groups
access_control:
claim_groups: groups
group_mapping:
trs-admins: admin
Entitlement-style groups¶
Some identity providers release groups as entitlements, i.e., URNs, in a
multi-valued claim (here entitlement) other than groups, and only via user
info:
security:
auth:
issuers:
- https://idp.example.org/
audience:
- <client ID of the portal>
userinfo_claims:
- entitlement
access_control:
claim_groups: entitlement
group_mapping:
"urn:example:group:trs-admins#issuer.example.org": admin
Settings for specific identity providers belong to the documentation of the deployment, not here.
Notes:
- The group (the URN above is illustrative) is a value of the claim. Quote
such keys in YAML, as they contain
:and#. - Without the claim in
userinfo_claims, no user gets the admin role. - Clients must request the scope that releases the claim when logging in
(here the one for
entitlement, besidesopenid), and the identity provider must allow it for the client. - TRS-Filer accepts only access tokens with the
typheaderat+jwt(security.auth.verify_token_type, enabled by default). Before going live, check that the identity provider issues such tokens to the clients, and that theiraudmatchesaudience: otherwise writes fail with401 Unauthorized. - To accept tokens of some clients only (e.g., of a portal or gateway in
front of TRS-Filer), list their client IDs in
security.auth.clients(FOCA compares them with theclient_idclaim of tokens, orazpwithout it); tokens of other clients, or without either claim, get401 Unauthorized. With several issuers, make sure that clients of one issuer can't claim the IDs of another's.
The policies (security.access_control.policies, see FOCA's access
control)
say what each role may do. They are named, so that an overlay can change,
add or remove single rules by name (FOCA's named
entries);
the base config has rules for reads only, which apply if login is required
for them (see Requiring login for
reads): writes are for admins.
An overlay that gives the policies as a list replaces all rules of the base
config instead; FOCA refuses to start if an overlay gives named rules after
one that gives a list.
The API specification (api.specs) is named trs, too, so that an overlay
can change it by name, e.g., to require login for
reads.
FOCA's service clients (security.auth.service_clients,
security.auth.on_behalf_of_header) are not supported; the app refuses to
start with them (see Authentication).
Local overrides¶
Docker Compose mounts
config.dev.yaml,
an overlay for local development that trusts a fictitious token issuer
(https://login.example.org/oidc/; set your own in a local override), sets
an example group claim (groups), accepts tokens for any audience and
requires no admins, so writes are refused until you set your identity
provider and admin group (see Getting started,
and Running without
authentication for
development without an identity provider). To override values locally
without changing it, put them in config.local.yaml and mount it in a
docker-compose.override.yaml (both are ignored by Git):
services:
trs-filer:
environment:
FOCA_CONFIG_OVERLAY: /etc/trs-filer/config.dev.yaml:/etc/trs-filer/config.local.yaml
volumes:
- ./config.local.yaml:/etc/trs-filer/config.local.yaml:ro
Public URL¶
The TRS API includes the URLs of tools and tool versions in its responses
(url), which TRS-Filer builds from custom.service:
<url_prefix>://<external_host>:<external_port>/<api_path>/tools/<id>. The
base config has the URL of the Docker Compose setup
(http://localhost:8080/ga4gh/trs/v2); deployments set their own, e.g.:
custom:
service:
url_prefix: https
external_host: trs.example.org
external_port: 443
api_path: ga4gh/trs/v2
The Helm chart sets it from its values (see Deployment);
other parts of the config go into its config value (see
Configuration there).
Identifiers¶
TRS-Filer generates the identifiers of tools, tool versions and tool classes
that are registered without one (POST): random strings of length
characters from charset, in custom.tool.id, custom.version.id and
custom.toolclass.id. By default, tool identifiers have six upper-case
letters and digits, and version and tool class identifiers six lower-case
letters and digits (tool classes created along with a tool, see Tool
classes, get identifiers like tools). charset lists the characters themselves (e.g.,
ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789); expressions such as
string.ascii_uppercase + string.digits are rejected on startup (see
Upgrading). Identifiers can also be chosen by
the client (PUT, see API).
Meta versions¶
Tools and versions have a meta_version (a field of the TRS API that tells
clients that an object changed), set in custom.tool.meta_version and
custom.version.meta_version:
- New tools and versions get the
initvalue (default 1). - Replacing a tool (
PUT /tools/{id}with the identifier of a stored tool) addscustom.tool.meta_version.increment(default 1) to its meta version, andcustom.version.meta_version.increment(default 1) to those of its versions that replace stored versions (same identifier); its other versions are new and get theinitvalue. - Replacing a version (
PUT /tools/{id}/versions/{version_id}) addscustom.version.meta_version.incrementto the version's meta version. - Adding, replacing or deleting a version with the version endpoints
(
POST /tools/{id}/versions,PUTandDELETE /tools/{id}/versions/{version_id}) also addscustom.tool.meta_version.incrementto the tool's meta version, as the tool as served (itsversions) changes. Setcustom.tool.meta_version.on_version_change: falseto change the tool's meta version only when the tool itself is replaced. Requests that fail change no meta version. - Stored meta versions that are not integers restart at the
initvalue.
Replacements with identical content increment the meta versions, too.
Concurrent changes of the same object may give it the same meta version,
or, for tools whose versions change at the same time, set it back to an
earlier value. Tool classes have none (custom.toolclass.meta_version is
unused).
Tool classes¶
Each tool has a tool class (e.g., workflow or command-line tool), which is
stored with the tool and in the list of tool classes (GET /toolClasses).
When a tool is registered or replaced (POST /tools, PUT /tools/{id}):
- If a tool class with the identifier of the tool's
toolclass.idexists (also if it is pending), the tool stores a copy of that tool class, not the one it was sent with, and the tool class is not changed. - Otherwise, if the user may add tool classes (see Adding tool
classes), the tool class is created as sent with
the tool (with a generated identifier if it has none); if not, the
registration fails with
400 Bad Requestand nothing is stored.
Tool classes are changed only with the tool class
endpoints. The copies stored with tools are not
updated when a tool class is replaced (PUT /toolClasses/{id}); register
the tools again to update them. The toolClass filter of GET /tools uses
the copies.
If tools with the same new tool class identifier but different tool classes are registered at the same time, the tool class of the first one is created, and the others store a copy of it, too.
Adding tool classes¶
custom.access_control.tool_classes sets who may add tool classes, with
the modes of the service types of Cloud
Registry:
open(the default): any authenticated user, explicitly (POST /toolClasses, orPUT /toolClasses/{id}with the identifier of no tool class) or by registering a tool with a new tool class (see above);controlled: only admins, both ways; tools of other users must have a registered tool class (400 Bad Requestotherwise), and theirPOST /toolClassesandPUT /toolClasses/{id}with new identifiers get403 Forbidden;moderated: any authenticated user, both ways, but the tool classes that users other than admins add are pending until an admin approves them (PUT /toolClasses/{id}/approval), and so are the tools that have them.
In every mode, only admins replace (PUT /toolClasses/{id} with the
identifier of a tool class), approve and delete tool classes; replacing a
pending tool class doesn't approve it. Tool classes store who added them
(internal field _createdBy, not served).
Pending tool classes and tools. In mode moderated, pending tool
classes are listed (GET /toolClasses) only to admins and the user who
added them, and pending tools are served (all reads of tools, also GET
/tools) only to admins and the users with a role on them (see Roles on
tools): the reads accept a token for
that (anonymous reads, and reads with an invalid token, see neither).
Others can still register tools with a pending tool class, whose tools are
pending too. GET /approvals lists the pending tool classes and tools that
the user may see (see Approving tool classes).
An admin's approval of a tool class approves all tools that have it. There is
no way back to pending: admins delete unwanted tool classes instead (DELETE
/toolClasses/{id}, once no tool has them). Pending tool classes stay
pending if the mode changes, and tools registered with them are pending
then too. Tools store whether they are pending (internal field _pending,
not served); the app sets it on registration, clears it on approval and
corrects it on startup. Without authentication (see Running without
authentication), there are
no admins: nothing is pending, and all tools are served.
Unused tool classes per user. Users other than admins may only have
added custom.access_control.max_unused_tool_classes tool classes that no
tool has (default: 10; null for no limit), as for the service types of
Cloud Registry (max_unused_service_types), in modes open and
moderated (in mode controlled, they can't add tool classes at all):
further tool classes are refused with 409 Conflict ("You have added 10
tool classes that no tool has, as many as allowed; ..."), whether added
explicitly or by registering a tool with a new tool class, until tools have
them or admins delete them (DELETE /toolClasses/{id}, see Access
control); pending and approved tool classes both
count. Tool classes become unused when their tools are deleted or change
their tool class, so the limit bounds how many tool classes each user can
add without tools (users can still leave more unused by registering tools
with new tool classes and deleting them, one registration per tool class).
Set 0 to let only admins add tool classes: users other than admins can
then only register tools with stored tool classes, as in mode
controlled. Tool classes stored before TRS-Filer recorded who added them
count for nobody. Concurrent requests of a user can exceed the limit by
their number. Without authentication, there are no users, and no limit.
GET /toolClasses is not paginated: the TRS specification defines no
paging for it, and the limit bounds how fast users can grow the list.
Policy rules. Keep the rule add-tool-classes
(security.access_control.policies) and set the mode instead of removing
the rule: without it, POST /toolClasses and PUT /toolClasses/{id} are
for admins only, but registering tools with new tool classes still adds
them, unless the mode is controlled. Without the rule for adding to a
collection (register-tools for tools, add-tool-classes for tool
classes), PUT with a new identifier is refused (403) also for users
whom other rules allow to replace objects (e.g., editors, see
Admins): such rules let them replace stored
objects, not add new ones.
For example, in a config overlay:
Deprecated: custom.toolclass.validation. custom.toolclass.validation:
true required stored tool classes for all tools; it now sets the mode
controlled, which replaces it, and the app warns of it on startup
(`custom.toolclass.validation` is deprecated ...). It is removed in
the next breaking release: set custom.access_control.tool_classes:
controlled instead. With the mode moderated, the app refuses to start
(the two contradict each other). Unlike before, admins add tool classes by
registering tools in mode controlled, and other users can't add tool
classes explicitly either (see Upgrading). The TRS
specification leaves who adds tool classes to registries (GET
/toolClasses lists "all tool-classes available"); in mode controlled,
every tool has a listed tool class, as with the old setting.
Remote files¶
Files of tool versions are registered with their content (content) or a
URL (url) that TRS-Filer retrieves the content from: on registration, and
when it serves the files of a version as a ZIP archive or a descriptor as
plain text (PLAIN_ types) and has no stored content for one of them.
custom.remote_files sets how URLs are retrieved:
| Option | Default | Description |
|---|---|---|
schemes |
[https] |
Allowed URL schemes (http, https). |
allowed_hosts |
[] |
Host names, IP addresses and CIDR networks that may be retrieved although they resolve to addresses that are not public (e.g., for local setups). |
connect_timeout |
5 |
Seconds to connect, per address of a host. |
read_timeout |
10 |
Seconds per read. |
total_timeout |
30 |
Seconds per file, including redirects. |
max_size |
1048576 |
Maximum size of a file, in bytes (1 MiB). |
max_redirects |
5 |
Redirects followed per file. |
Hosts must resolve to public addresses only (also after redirects), unless
they are listed in allowed_hosts; proxy settings of the environment are
not used. Retrievals that fail (e.g., a refused URL, a timeout, a response
other than 2xx or a file that is too large) fail the registration with
400 Bad Request; for ZIP archives and plain descriptors, 404 Not
Found. The app logs the reason, which names the problem but not the URL
(e.g., InvalidURL: The URL has user info; for errors of lower layers,
only their types, such as ConnectionError (caused by MaxRetryError,
NameResolutionError, gaierror)), next to the URL without its user info.
Name resolution is not limited by the timeouts. For example, to register
files of a web server on the Docker Compose network in a local setup:
I/O Schemas¶
custom.io_schema.validation (experimental, false by default) checks
GA4GH I/O Schema files (files of type OTHER named ga4gh-io.schema.json)
on registration against the meta-schema of the GA4GH I/O Profile 0.3.0,
and rejects invalid ones with 400 Bad Request (see Checking I/O
Schemas); custom.io_schema.timeout
(default: 10) is the time limit of checking a file, in seconds:
File storage¶
By default, TRS-Filer stores the content of files in MongoDB, with the
other data of tools. Optionally, it stores the content of files in
S3-compatible object storage instead, and keeps only their metadata in
MongoDB (custom.storage):
| Option | Default | Description |
|---|---|---|
backend |
mongodb |
Where the content of files registered from now on is stored: mongodb or s3. |
s3.bucket |
Name of the bucket, which must exist; required if s3 is set. |
|
s3.endpoint_url |
URL of the S3 API, e.g., https://s3.example.org; boto3's default if not set (AWS, or the environment variable AWS_ENDPOINT_URL_S3 or AWS_ENDPOINT_URL). |
|
s3.region |
Region of the bucket; boto3's default if not set (e.g., the environment variable AWS_REGION). |
|
s3.prefix |
"" |
Prefix of the keys of the objects (e.g., trs-filer/), so that a bucket can be shared. |
s3.addressing_style |
path |
path (<endpoint>/<bucket>/<key>, which most self-hosted servers need), virtual (<bucket>.<endpoint>/<key>) or auto (boto3's choice). |
s3.presigned_urls |
false |
Serve files via presigned URLs of their objects instead of through the app (see below). |
s3.presigned_url_expiry |
300 |
Validity of presigned URLs, in seconds (at most 604800, 7 days). |
s3.connect_timeout |
5 |
Seconds to connect to the S3 API. |
s3.read_timeout |
10 |
Seconds per read from the S3 API. |
TRS-Filer talks to the storage through the S3 API only (with boto3), so any S3-compatible server or service works, e.g., SeaweedFS or Zenko CloudServer (both Apache-2.0), Ceph RGW, or the S3 services of cloud and HPC providers. CI tests with SeaweedFS.
Credentials are never part of the config (which rejects unknown
settings): boto3 takes them from the environment with its default
credential
chain,
e.g., from the environment variables AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY (set from a Secret, see Object
storage for the Helm chart), or a web
identity token. The app needs the S3 permissions to put, get and delete
objects in the bucket (with the prefix, if set); the bucket isn't created
by the app.
For example:
custom:
storage:
backend: s3
s3:
endpoint_url: https://s3.example.org
bucket: trs-filer
region: us-east-1
prefix: trs-filer/
How files are stored:
- When a version is registered (also with its tool, and when it is
replaced), the content of each of its files, once all files are valid
(and retrieved from their URLs, see Remote files), is
written once as a new object, under a random key with the prefix (as
text/plain; charset=utf-8). MongoDB keeps the file's metadata and a reference to the object, with the SHA-256 checksum and the size of the content; the reference is never served. - When files are served, their content is read from the object and checked
against the checksum and size; changed or missing objects fail the
request with
500 Internal Server Error, and the app logs the key. A missing object is first read again from the stored files once, as the version may have been replaced or deleted meanwhile (this needs reads from the primary: with areadPreferenceof secondaries inMONGO_URI, the retry may still see the old files). - Objects that no stored file refers to any more are deleted at the end of
the request that released them: those of replaced and deleted versions
and tools, and those of registrations that failed. If deleting an object
fails, or the app stops during a request, objects can be left without a
file that refers to them (logged at
ERROR); they are not served. - The backend only decides where new files go: files stay where they were
stored. With
backend: mongodbands3set, files in object storage are still served (e.g., to switch back); withouts3, they fail with500 Internal Server Error. Existing files are not moved to object storage.
With presigned URLs (s3.presigned_urls: true), clients get files from
the object storage directly, which then must be reachable by them: JSON
responses with files (descriptors, tests, containerfiles) have the
presigned URL of the object as url, instead of the content and of the
URL given on registration, and the SHA-256 checksum of the content
(sha-256, unless given on registration); plain responses (PLAIN_
types) redirect to the presigned URL (302 Found). Presigned URLs serve
the content as text/plain and as a download (Content-Disposition:
attachment), so that browsers don't display it. Presigned URLs expire
after s3.presigned_url_expiry seconds, or earlier, when the version is
replaced or deleted (its objects are deleted). ZIP archives are still built
by the app.
A presigned URL lets anybody who has it read the file until it expires, without a token, also if the deployment requires login for reads (see Requiring login for reads): keep the expiry short there, and keep the URLs out of logs. Serve the S3 API on another host than TRS-Filer, so that clients don't send their access tokens along with the redirect.
Docker Compose doesn't bundle an object store. To try object storage
locally, add SeaweedFS (here with the S3 API without authentication, for
tests only) in a docker-compose.override.yaml, and set the backend in a
config overlay (e.g., config.s3.yaml, after config.dev.yaml):
# docker-compose.override.yaml
services:
trs-filer:
environment:
FOCA_CONFIG_OVERLAY: /etc/trs-filer/config.dev.yaml:/etc/trs-filer/config.s3.yaml
# Any values: SeaweedFS without authentication doesn't check them
AWS_ACCESS_KEY_ID: test
AWS_SECRET_ACCESS_KEY: test
volumes:
- ./config.s3.yaml:/etc/trs-filer/config.s3.yaml:ro
seaweedfs:
image: docker.io/chrislusf/seaweedfs:4.51
command: server -s3 -ip.bind=0.0.0.0
ports:
- "127.0.0.1:8333:8333"
# config.s3.yaml
custom:
storage:
backend: s3
s3:
endpoint_url: http://seaweedfs:8333
bucket: trs-filer
region: us-east-1
Then create the bucket, e.g., with the AWS CLI on the host:
aws --endpoint-url http://127.0.0.1:8333 s3 mb s3://trs-filer.
Request size limit¶
Request bodies are limited to server.max_content_length bytes (default:
1048576, i.e., 1 MiB), by FOCA: larger requests, also chunked ones, are
rejected with 413 Content Too Large before they are parsed. The app
refuses to start without a limit. Raise it to register tools with larger
inline file contents, together with the limits of any proxies in front of
the app; prefer files with URLs.
Rate limits¶
TRS-Filer has no rate limits of its own. In production, limit the rate of requests per client at the reverse proxy or ingress controller in front of it, so that single clients can't overload the app or its database.
Server¶
The image serves the app with Gunicorn, with the
settings of
trs_filer/gunicorn_config.py:
it listens on port 8080, with one worker process with four threads.
Override settings with the environment variable GUNICORN_CMD_ARGS (e.g.,
--workers 2 --threads 8), but not --bind: the image's health check and
the Helm chart expect port 8080. Behind a reverse proxy, set its addresses
in FORWARDED_ALLOW_IPS, so that Gunicorn trusts its X-Forwarded-*
headers (with the Helm chart, in trsFiler.extraEnv, see Forwarded
headers). server.host and
server.port of the config only apply to Flask's development server
(python -m trs_filer.app). server.debug must stay off outside local
development: Flask's debugger allows executing arbitrary code.
Database¶
TRS-Filer stores tools (with their versions and files; the content of files
optionally in object storage, see File storage) and tool
classes in the collections tools and toolclasses of the MongoDB
database trsStore at db.host and db.port (in the base config:
mongodb:27017, the service of Docker Compose). FOCA reads the environment variables MONGO_HOST,
MONGO_PORT, MONGO_DBNAME, MONGO_USERNAME and MONGO_PASSWORD, or a
complete connection string in MONGO_URI, which take precedence over the
config (see the FOCA documentation);
with MONGO_DBNAME set, the app uses that database instead of trsStore.
FOCA keeps the rules of access control in the collection policy_rules of
the same database (security.access_control.db_name and
collection_name), and its changes in policy_rules_changes.
Service info¶
GET /service-info is served by FOCA from section service_info of the
config, according to the GA4GH service info schema (validated on startup).
Its version is the version of the installed package
(service_info.package). The base config describes the app, not a
deployment: its id (org.example.trs-filer) and organization are
placeholders in reserved example domains (RFC 2606). FOCA warns on startup
while id, organization.url, contactUrl or documentationUrl are in
such a domain (as in the example below), not of other fields (e.g.,
organization.name, which overlays keep unless they set it). Deployments
set their own id, name, description, organization, contactUrl and
environment in service_info.service, e.g.:
service_info:
service:
id: org.example.trs
name: Example TRS
description: Tools and workflows of the Example Organization.
organization:
name: Example Organization
url: https://example.org
contactUrl: mailto:trs-admins@example.org
environment: prod
The Helm chart has no value for the service info yet, so chart deployments serve the placeholders unless they deploy an overlay of their own (see Configuration in the deployment docs).
Errors¶
Errors are problem details, sent as application/json (section
exceptions, see Errors). Requests that the validation of
the API specification rejects (e.g., an unknown descriptor type) get the
reason as detail, shortened to exceptions.max_detail_length characters
(500 by default); set exceptions.validation_details: false to send a
generic detail instead (see FOCA's
documentation).
The base config sets exceptions.validation_reasons: true, so that this
detail is the validator's reason, as up to FOCA 0.21, which may quote
the rejected values; set it to false for FOCA's default, which names
the invalid fields by location and code instead.
Requests that TRS-Filer itself refuses (e.g., deleting the last version of
a tool) get their reason as detail whatever exceptions.validation_details
is set to; it names the rule, not the values sent (see
Errors). Like all details, it is shortened to
exceptions.max_detail_length characters, and not sent if
exceptions.public_members doesn't list detail.