Upgrading¶
The release notes list the breaking changes of each release
and how to upgrade. This page describes how existing setups and data are
affected by the changes since the tag v0.3.0 (2023) and the images
published from dev before the move to FOCA 0.17 (tagged latest and by
date, with FOCA 0.12); for images from dev since then, see Images from
dev since FOCA 0.17.
Configuration¶
TRS-Filer runs on FOCA 0.19 and Python 3.13. Custom configs must apply the
config changes of FOCA 0.14 to 0.19 (see the FOCA release
notes and migration
guides), e.g., exceptions.mapping instead of the old exception members,
no server.environment or server.use_reloader, and no top-level
plugins section other than FOCA's (validated since FOCA 0.18, with
unknown keys refused; TRS-Filer loads no plugins);
server.max_content_length must be set (see Request size
limit). Configs are best written as
overlays of the base config (see Configuration), which
the image no longer has at /app/trs_filer/config.yaml (the package is
installed, not copied).
- The identifier character sets (
custom.tool.id.charset,custom.version.id.charset,custom.toolclass.id.charset) list the characters themselves (e.g.,ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789); expressions such asstring.ascii_uppercase + string.digitsare rejected on startup. - Identifiers of tool classes registered with the tool class endpoints
follow
custom.toolclass.id(they usedcustom.tool.id): by default lower-case letters and digits. custom.service(the public URL) defaults tohttp://localhost:8080; deployments set their own.- File contents are retrieved from URLs as configured in
custom.remote_files(see Remote files):httpURLs are only retrieved ifhttpis added toschemes, hosts that resolve to addresses that are not public only if listed inallowed_hosts, and responses other than2xxor larger thanmax_sizefail the registration with400 Bad Request(before, error pages were stored as content).
Service info¶
POST /service-info and the service_info collection are removed: the
service info is configured in service_info.service (FOCA's section, see
Service info); custom.service_info is
rejected on startup. The collection stays in existing databases, unused;
drop it if you like.
API¶
GET /tools:limit=0gave all tools; it now gives400 Bad Request, as do other limits below 1. Invalidoffsetandlimitvalues give400instead of422.limitis at most 1000 (larger values are reduced to 1000).next_pageis omitted on the last page, andlast_pageis counted in steps oflimitfrom the requested offset. Filters of versions and images apply to the same version and image (they overwrote each other). The linksnext_pageandlast_pagekeep the filters of the request (they had onlyoffsetandlimit). An emptyauthormatches only empty authors (it matched all tools), and the unsupported filtertagsgives400 Bad Request(it was ignored). See Listing tools.- Registering or replacing a tool no longer changes an existing tool class:
the tool stores a copy of the stored tool class instead of the one it was
sent with. With
custom.toolclass.validation: true, a tool with an unknown tool class is no longer stored (before, it was stored although the request failed with400). See Tool classes. - Replacing a tool, version or tool class with an identical one succeeds
(it failed with
500). - Replacing tools and versions increments their
meta_version(it stayed at the initial value), as configured incustom.tool.meta_version.incrementandcustom.version.meta_version.increment. Adding, replacing or deleting a version also increments themeta_versionof its tool, so clients see that the tool changed; setcustom.tool.meta_version.on_version_change: falseto keep the tool's meta version unchanged on version changes. See Meta versions. - Deleting the last version of a tool gives
400 Bad Request; delete the tool instead. - Versions without
is_productionare registered as non-production versions (they failed with500).
Errors¶
Errors are problem details (title, status, detail), sent as
application/json, instead of objects with message and code (see
Errors). Requests that the validation of the API
specification rejects get the reason as detail (e.g., 'FOO' is not one
of ['CWL', ...]), no longer the generic The request is malformed.; set
exceptions.validation_details: false to keep the generic one (see
Errors).
Database¶
Without MONGO_DBNAME, the image uses the database trsStore (as
configured) instead of trs-filer-db. Deployments with data in
trs-filer-db (e.g., the Docker Compose setup of earlier versions) set
MONGO_DBNAME=trs-filer-db, or move the data (see below); the Helm chart
sets it already (value mongodb.dbName, by default trs-filer-db).
MongoDB 4.4 or later is required (Docker Compose and the Helm chart use
7.0). MongoDB 7.0 can't open data files of 3.6, which earlier Compose
setups used: migrate them with mongodump (with MongoDB 3.6) and
mongorestore (with 7.0), e.g., into trsStore, or upgrade one major
version at a time. For the Helm chart, see Helm chart.
Image¶
- Image tags:
latestis the highest stable release from the first release with the new release process on, no longer the head ofdev, which is taggeddev(andsha-<commit>per commit); dated tags are no longer published. Pin a release (X.Y.Z) in deployments (see Versioning and releases). Images published by the new workflows are scanned, attested and signed (see Verifying images and charts). - Gunicorn is configured by
trs_filer/gunicorn_config.pyand the environment variableGUNICORN_CMD_ARGS(e.g.,--workers 2; see Server);GUNICORN_PROCESSES,GUNICORN_THREADSandTRS_CONFIGare no longer read, and Gunicorn always binds0.0.0.0:8080. - Forwarded headers (
X-Forwarded-*) are trusted from Gunicorn's default (127.0.0.1) only, no longer from every client: behind a proxy that terminates TLS, setFORWARDED_ALLOW_IPSto its addresses (the Helm chart sets it). - The image runs as user 1000 (also with a read-only root file system and a
writable
/tmp) and reports its version (e.g., in the service info) from the build argumentSETUPTOOLS_SCM_PRETEND_VERSION.
Docker Compose¶
The service is named trs-filer (was trs), and the app is published at
http://localhost:8080/ga4gh/trs/v2/ on the loopback interface only (was
port 80 on all interfaces). MongoDB data lives in the named volume
mongodb-data instead of ../data/trs/db: migrate it with mongodump and
mongorestore if needed (see Database). The Compose setup
applies the overlay config.dev.yaml (see Local
overrides).
Helm chart¶
Releases attach the chart (see Verifying images and
charts); upgrade with it, and
your values file, as described in Deployment.
Releases installed from the chart in the repository before (from the tag
v0.3.0 or from dev) may need more steps, depending on two things, which
you check before the upgrade:
- The name of MongoDB's claim (
kubectl get pvc -n <namespace>): the chart calls MongoDB's Deployment, Service and PersistentVolumeClaimmongodb-<appName>(trs_filer.appName, by defaulttrs-filer); charts before July 2024, including that ofv0.3.0, called themmongodb. Helm deletes resources that are no longer in the chart, so upgrading such a release deletes the claimmongodband, with the reclaim policyDelete(the default of most storage classes), its volume and the data, and starts MongoDB on a new, empty claim. -
The running MongoDB version: the chart runs MongoDB 7.0 (
mongodb.image, see MongoDB). Charts before January 2026 ranmongo:3.6, whose data files 7.0 can't open; the chart ondevsince then ran the floatingmongo:noble(MongoDB 8.2, 8.3 or 9.0, depending on when the pod last started), whose data 7.0 can't open either (MongoDB doesn't support downgrades). Check the version (and the feature compatibility version) of the running server (withmongosh, ormongo, the old shell, in images before 6.0);OLDis the MongoDB Deployment (mongodbormongodb-<appName>):kubectl exec deploy/OLD -n <namespace> -- sh -c \ '"$(command -v mongosh || echo mongo)" --quiet --eval "printjson({version: db.version(), fcv: db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1}).featureCompatibilityVersion})"'If MongoDB doesn't start any more (
CrashLoopBackOff, e.g., after a restart pulled a newermongo:noblethat can't open the data), its log names the versions (kubectl logs deploy/OLD -n <namespace> --previous): set an image that can open the data (e.g.,kubectl set image deploy/OLD OLD=docker.io/library/mongo:8.3 -n <namespace>; the container is named after the Deployment), wait until it runs, then continue here.
Then:
- Claim
mongodb-<appName>and MongoDB 7.0: upgrade as usual. - Claim
mongodb-<appName>and MongoDB 8.x or 9.0: setmongodb.imagein the values file to the running major and minor version (e.g.,docker.io/library/mongo:8.2, asdb.version()reports it; mind the kernel requirements of 8.x, see MongoDB) and upgrade; or move the data to 7.0 by dump and restore as below. - Otherwise (claim
mongodb, or MongoDB 3.6): dump the data before the upgrade and restore it afterwards, as below.
Dump and restore¶
The app is down from the first to the last step. The commands are for the
release trs-filer with the default trs_filer.appName (trs-filer, the
name of the app's Deployment and the label app of its pods; MongoDB's
Deployment and claim are mongodb-trs-filer) and database
(trs-filer-db, mongodb.dbName); replace them if yours differ. OLD is
the MongoDB Deployment, and claim, before the upgrade (mongodb or
mongodb-trs-filer). Clients can reach the app whenever it runs, also with
the empty database between the upgrade (step 4, which starts the app again)
and step 5, and the restore replaces what they write then: keep clients
away during the migration (e.g., by blocking the host at a proxy in front
of the cluster, or in a maintenance window).
-
Stop the app and wait until its pod is gone, so that the data doesn't change during the dump:
-
Count the documents of each collection, for a comparison after the restore, and dump the database to a local file; check that
mongodumpsucceeds (the command printsDump OK):kubectl exec deploy/OLD -n <namespace> -- sh -c \ '"$(command -v mongosh || echo mongo)" --quiet trs-filer-db --eval "db.getCollectionNames().forEach(c => print(c, db[c].countDocuments ? db[c].countDocuments({}) : db[c].count()))"' kubectl exec deploy/OLD -n <namespace> -- \ mongodump --quiet --db trs-filer-db --archive > trs-filer-db.archive \ && echo "Dump OK" -
Keep the old volume as a fallback: set the reclaim policy of its PersistentVolume to
Retain, which keeps the volume (with the data) when its claim is deleted:kubectl patch pv "$(kubectl get pvc OLD -n <namespace> \ -o jsonpath='{.spec.volumeName}')" \ -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'- With the claim
mongodb, also annotate the claim, so that Helm keeps it (Helm checks the annotation on the claim in the cluster before deleting it on upgrades):kubectl annotate pvc mongodb -n <namespace> helm.sh/resource-policy=keep. - With the claim
mongodb-trs-filer(and MongoDB 3.6, or a version to leave), the upgrade keeps the claim, so MongoDB would start on the old data files; delete MongoDB's Deployment and the claim (the volume is retained), so that the upgrade creates them anew, with an empty volume:kubectl delete deploy/mongodb-trs-filer pvc/mongodb-trs-filer -n <namespace>.
- With the claim
-
Update the values file (see Values) and upgrade the release with the chart of the release, e.g.:
The chart of the tag
v0.3.0deploys the app with the APIextensions/v1beta1, which Kubernetes no longer serves since 1.16; ifhelm upgradefails because the release's manifest contains removed APIs, map them first with the mapkubeapis plugin (helm mapkubeapis trs-filer -n <namespace>). -
Stop the app again, so that it doesn't write during the restore, and wait until MongoDB has initialized the new volume (its entrypoint first runs a temporary server for the init script, then starts MongoDB) and accepts connections:
kubectl scale deploy/trs-filer -n <namespace> --replicas=0 kubectl wait pod -l app=trs-filer -n <namespace> --for=delete --timeout=120s timeout 300 sh -c 'until kubectl logs deploy/mongodb-trs-filer -n <namespace> \ | grep -q "ready for start up"; do sleep 2; done' \ && timeout 120 sh -c 'until kubectl exec deploy/mongodb-trs-filer -n <namespace> -- \ mongosh --quiet --eval "db.runCommand({ping: 1}).ok"; do sleep 2; done' \ && echo "MongoDB ready"Continue only if
MongoDB readyis printed; otherwise check MongoDB's pod and log (kubectl get pods,kubectl logs): e.g., after a failed init, MongoDB restarts without it and doesn't print the line. -
Restore the dump, replacing the collections that the app has created, and count the documents again:
kubectl exec -i deploy/mongodb-trs-filer -n <namespace> -- \ mongorestore --quiet --drop --archive < trs-filer-db.archive \ && kubectl exec deploy/mongodb-trs-filer -n <namespace> -- \ mongosh --quiet trs-filer-db --eval 'db.getCollectionNames().forEach(c => print(c, db[c].countDocuments ? db[c].countDocuments({}) : db[c].count()))'If the restore fails on the options of an index, restore with
--noIndexRestore: the app creates its indexes on startup. The restore is repeatable (--drop). -
Only if the restore succeeded and the counts match those of step 2, start the app:
Once the new release works, delete the old claim mongodb (if any:
kubectl delete pvc mongodb -n <namespace>), the retained volume
(kubectl delete pv <name>; this deletes only the PersistentVolume object:
delete its storage, e.g., the disk, with your storage provider) and the
local dump. The service_info
collection of the dump is no longer used (see Service
info).
Values¶
Since the chart of v0.3.0, the values changed:
clusterTypeis removed: the chart creates an Ingress (trs_filer.ingress.enabled, by defaulttrue; HTTPS via cert-manager withtrs_filer.ingress.https) or, with the Ingress disabled, an OpenShift Route (with TLS at the router), asv0.3.0did withclusterType: openshift.trs_filer.route.enabled(by defaulttrue) turns the Route off, e.g., for a Route of your own, andtrs_filer.route.annotationssets its annotations (see Ingress and Route).host_namehas no default (it was a test host); set it.trs_filer.imageiselixircloud/trs-filer(released charts set the release's image), no longerlvarin/trs-filer:0.1.0.trs_filer.resourcesandmongodb.resourcesset the requests and limits of the app and of MongoDB.mongodb.imageisdocker.io/library/mongo:7.0(wasmongo:3.6, andmongo:nobleondev), pulled asmongodb.imagePullPolicysays;mongodb.dbNamesets the app's database (trs-filer-dbif empty).
The chart annotates MongoDB's claim with helm.sh/resource-policy: keep
(see MongoDB), and replaces MongoDB's pod with the
Recreate strategy.
MongoDB's init script (ConfigMap mongo-init-script-<appName>, was
mongo-init-script) creates the collections of the app's database only
(no service_info, and nothing in trsStore); it runs on an empty volume
only, so existing databases are not changed. The app gets the MongoDB host
from MONGO_HOST (the Service mongodb-<appName>), also if the config Job
fails. The chart in the repository has the placeholder version 0.0.0
(was 2.0.0); released charts have the release's version.
Images from dev since FOCA 0.17¶
Images from dev built since the move to FOCA 0.17 (dev, sha-<commit>)
ran on an unreleased FOCA commit; releases run on FOCA 0.19. Besides the
detail of rejected requests (see Errors), the startup warning
about the placeholders of the service info changes: FOCA logs it
(logger foca.api.service_info) while id, organization.url,
contactUrl or documentationUrl are in a reserved example domain (RFC
2606, e.g., org.example.trs or https://trs.example.org), instead of the
app (logger trs_filer.app), which warned of the placeholders id and
organization (name or URL) of the base config only; a placeholder
organization.name is no longer warned of. FOCA's plugin store
(collection plugin_store) and plugin listing are for gateway apps:
TRS-Filer creates no collections or indexes beyond its own, so its
database user needs no new rights.