0.4.0
This is the first release of TRS-Filer since v0.3.0 (2023), and the first
with a release image that is scanned for vulnerabilities, attested and
signed, and with an attested and signed Helm chart.
It moves the app to FOCA 0.19 and Python 3.13, hardens the retrieval of
remote files, fixes listing and registration bugs, adds zip downloads of a
version's files and reworks the Helm chart, the image and the Docker Compose
setup. Many of these changes are breaking (see below) for configs, clients,
images and existing chart installs: read the upgrade
steps before
upgrading, in particular those for the Helm
chart,
as upgrading a chart install from v0.3.0 can delete MongoDB's data. See
the documentation for
details.
Breaking changes¶
- Config: TRS-Filer runs on FOCA 0.19 (from 0.12) and Python 3.13. Custom
configs must apply FOCA's config changes of 0.14 to 0.19 (e.g.,
exceptions.mapping, noserver.environment), setserver.max_content_length(1 MiB in the base config), and give the identifier character sets (custom.*.id.charset) as the characters themselves, not as expressions;custom.servicedefaults tohttp://localhost:8080(#88, #139; #187, #190) - Service info:
POST /service-infois removed, and the app no longer uses theservice_infocollection (it stays in existing databases); the service info is configured in FOCA'sservice_info.service(custom.service_infois rejected on startup), with placeholders that FOCA warns about (#88, #139) - Errors: errors are problem details (
title,status,detail) instead ofmessageandcode; requests that the validation of the API specification rejects get the reason asdetail(exceptions.validation_details: falsekeeps the generic text) (#88, #139; #187, #190) - Database: without
MONGO_DBNAME, the app uses the databasetrsStore(wastrs-filer-db), and MongoDB 4.4 or later is required; MongoDB 7.0 (Docker Compose, Helm chart) can't open the 3.6 data of earlier setups, so migrate it by dump and restore (#88, #139) - Remote files: file contents are retrieved from
httpsURLs only by default, never from hosts that resolve to non-public addresses unless allowed, and failing or oversized responses fail the registration with400(they were stored as content); seecustom.remote_files(#140, #142) - API:
GET /toolsrefuseslimit=0(it gave all tools) and invalidoffsetandlimitvalues with400(was422), capslimitat 1000, omitsnext_pageon the last page, applies the version and image filters to the same version and image, matches an emptyauthoronly to empty authors and refuses the unsupported filtertagswith400(#126, #172; #179, #181) - API: registering a tool no longer changes an existing tool class (the
tool stores a copy of the stored one); with
custom.toolclass.validation, tools with unknown tool classes are no longer stored; identifiers of tool classes followcustom.toolclass.id(wascustom.tool.id); deleting the last version of a tool gives400(#126, #172; #88, #139) - API: replacing tools and versions increments their
meta_version, and adding, replacing or deleting a version increments its tool's (custom.tool.meta_version.on_version_change: falseturns the latter off) (#126, #172; #176, #185) - Image: the image runs as user 1000 (also with a read-only root file
system), installs the app as a package (the base config is no longer at
/app/trs_filer/config.yaml; use an overlay viaFOCA_CONFIG_OVERLAY), configures Gunicorn viaGUNICORN_CMD_ARGS(GUNICORN_PROCESSES,GUNICORN_THREADSandTRS_CONFIGare no longer read), and trusts forwarded headers from127.0.0.1only (setFORWARDED_ALLOW_IPSbehind a proxy) (#111, #123; #143) - Image tags:
latestis the highest stable release (it was the head ofdev, now taggeddevandsha-<commit>); dated tags are no longer published. Pin a release in deployments (#107, #133; #174) - Docker Compose: the service is
trs-filer(wastrs), published on127.0.0.1:8080(was port 80 on all interfaces), with MongoDB 7.0 and its data in the named volumemongodb-data(was../data/trs/db) (#111, #123; #143) - Helm chart: MongoDB's Deployment, Service and claim are named
mongodb-<appName>(weremongodb), so upgrading a chart install fromv0.3.0deletes the old claim and, usually, its data; MongoDB is pinned tomongo:7.0(wasmongo:3.6inv0.3.0, the floatingmongo:nobleondev). Follow the upgrade steps, or the database is lost (#114; #121; #197, #205) - Helm chart: the app's memory request and limit are now 256Mi and 384Mi
(were 128Mi each; QoS class Burstable instead of Guaranteed). Pods may no
longer be scheduled on clusters or namespaces with tight memory quotas;
lower
trs_filer.resourcesthere, as described in the chart's README (#200, #206) - Helm chart: values changed:
clusterTypeis removed; the chart creates an Ingress iftrs_filer.ingress.enabled(by defaulttrue), otherwise an OpenShift Route where the cluster has the Route API. The image iselixircloud/trs-filer, andhost_nameis empty by default (#97; #112; #116; #121)
Features¶
- API:
GET /tools/{id}/versions/{version_id}/{type}/files?format=zipserves the files of a version as a zip archive (#72, #119) - Config:
custom.tool.meta_version.on_version_changeand the increments of meta versions (custom.tool.meta_version.increment,custom.version.meta_version.increment) (#126, #172; #176, #185) - Helm chart: a Kubernetes Ingress next to the OpenShift Route, the config
Job sets MongoDB's host to the chart's MongoDB Service,
appNameon all resources (so that several releases fit in one namespace), MongoDB's database name as a value (mongodb.dbName), resource requests and limits, and a MongoDB init script (#97; #112; #113; #114; #121) - Helm chart:
trs_filer.ingress.classNameandtrs_filer.ingress.annotations, e.g., to restrict write methods at the ingress controller (#198, #204) - Helm chart:
trs_filer.route.annotations, e.g., to allow only trusted clients at the OpenShift router, andtrs_filer.route.enabled(by defaulttrue), to turn the Route off, e.g., for a Route of your own; NOTES.txt points to the planned authentication (#7) and, if the chart exposes nothing, gives akubectl port-forwardhint (#210, #213) - Helm chart:
mongodb.imagePullPolicy; MongoDB's claim is kept on uninstall (helm.sh/resource-policy: keep), and MongoDB uses theRecreatestrategy (#197, #205) - Releases: the release image is scanned, attested and signed, and the
release has the packaged Helm chart (
trs-filer-X.Y.Z.tgz) with its Sigstore bundle; images ofdevare scanned, attested and signed too (#107, #133; #174; #134, #180; #184, #186)
Security¶
- Remote files: the retrieval of file contents from URLs (on registration and for zip downloads) allows only the configured schemes and hosts with public addresses (checked on every redirect), with limits on size, time and redirects, against server-side request forgery (#140, #142)
- Image: base image pinned by digest, non-root user, an allowlist
.dockerignore, and forwarded headers trusted from127.0.0.1only by default (#111, #123; #143) - Helm chart: warnings in the chart's README and NOTES.txt that the API has no authentication, with a method-restriction example for ingress-nginx; see also the new page "Access control" of the documentation (#198, #204; #175, #178)
Fixes¶
- API: production versions need checksums for all files, other versions don't (the check was applied to all versions) (#118)
- API: replacing a tool, version or tool class with an identical one
succeeds (it failed with
500); versions withoutis_productionare registered as non-production versions (they failed with500);DELETE /toolClasses/{id}checks all tools, not only the first 1000; retries of generated identifiers use new identifiers (#126, #148, #172) - API: the page links
next_pageandlast_pagekeep the filters of the request, and concurrent registrations of tools with the same new tool class store the same tool class (#179, #181) - Image: the image starts the server by default (it exited like a one-off job) (#117)
- Helm chart: NOTES.txt prints
httporhttpsURLs to match how the app is exposed, warns when nothing exposes it, and no longer uses the deprecated "Elixir Cloud" name (#189, #198, #204) - Helm chart: the app gets
MONGO_HOSTdirectly, so it reaches MongoDB even if the config Job fails; the init script creates only the app database's collections, under a ConfigMap name scoped byappName(#202, #207)
Documentation¶
- A documentation site (MkDocs Material, on Read the Docs) with getting started, configuration, API, access control, deployment, operations, upgrading, development and API reference pages (#129, #170; #175, #178; #130, #138)
- Upgrade guide for configs, clients, images, Docker Compose and Helm chart
installs from
v0.3.0anddev; MongoDB version and persistence in the chart's README (#197, #205) - The Helm chart can't set the app config yet (the service info stays a placeholder), with a workaround until #135 (#199, #203)
- Memory sizing of the app (#200, #206); TRS version labels explained on the page "API" (#202, #207)
- Citation metadata (
CITATION.cff, archived on Zenodo with each release) (#132, #171) - Community files and security policy of the organisation, a new logo and an updated README (#131, #137; #157, #158; #188, #194; #182, #183)
Maintenance¶
- Packaging with
pyproject.tomlonly (version from Git tags via setuptools-scm);setup.py,setup.cfgandrequirements*.txtare removed (#111, #123; #143) - Tests: an app test harness (mongomock or a real MongoDB), typed tests and 100% coverage required (#125, #169; #106, #147, #150; #127, #173)
- Linting and formatting with ruff, mypy, pre-commit, ast-grep and semgrep, shared with the other repositories of the organisation (#124, #145; #146, #149; #153, #154)
- CI: workflow and job names aligned with the organisation, a single gate check, forks supported, shared actions pinned, and the Docker Hub organisation as a variable (#155, #156; #159, #160; #161, #162; #163, #164; #165, #168; #191, #195)
- Removed the unused 2020 TRS specifications; the chart in the repository
has a placeholder version (
0.0.0,appVersion: dev) and a description (#202, #207) - Dependency updates (#115; #151; #152)