Skip to content

Upgrading

The release notes list the breaking changes of each release and how to upgrade. This page describes how existing setups and data are affected by the changes since the tag v0.3.0 (2023) and the images published from dev before the move to FOCA 0.17 (tagged latest and by date, with FOCA 0.12); for images from dev since then, see Images from dev since FOCA 0.17.

Configuration

TRS-Filer runs on FOCA 0.19 and Python 3.13. Custom configs must apply the config changes of FOCA 0.14 to 0.19 (see the FOCA release notes and migration guides), e.g., exceptions.mapping instead of the old exception members, no server.environment or server.use_reloader, and no top-level plugins section other than FOCA's (validated since FOCA 0.18, with unknown keys refused; TRS-Filer loads no plugins); server.max_content_length must be set (see Request size limit). Configs are best written as overlays of the base config (see Configuration), which the image no longer has at /app/trs_filer/config.yaml (the package is installed, not copied).

  • The identifier character sets (custom.tool.id.charset, custom.version.id.charset, custom.toolclass.id.charset) list the characters themselves (e.g., ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789); expressions such as string.ascii_uppercase + string.digits are rejected on startup.
  • Identifiers of tool classes registered with the tool class endpoints follow custom.toolclass.id (they used custom.tool.id): by default lower-case letters and digits.
  • custom.service (the public URL) defaults to http://localhost:8080; deployments set their own.
  • File contents are retrieved from URLs as configured in custom.remote_files (see Remote files): http URLs are only retrieved if http is added to schemes, hosts that resolve to addresses that are not public only if listed in allowed_hosts, and responses other than 2xx or larger than max_size fail the registration with 400 Bad Request (before, error pages were stored as content).

Service info

POST /service-info and the service_info collection are removed: the service info is configured in service_info.service (FOCA's section, see Service info); custom.service_info is rejected on startup. The collection stays in existing databases, unused; drop it if you like.

API

  • GET /tools: limit=0 gave all tools; it now gives 400 Bad Request, as do other limits below 1. Invalid offset and limit values give 400 instead of 422. limit is at most 1000 (larger values are reduced to 1000). next_page is omitted on the last page, and last_page is counted in steps of limit from the requested offset. Filters of versions and images apply to the same version and image (they overwrote each other). The links next_page and last_page keep the filters of the request (they had only offset and limit). An empty author matches only empty authors (it matched all tools), and the unsupported filter tags gives 400 Bad Request (it was ignored). See Listing tools.
  • Registering or replacing a tool no longer changes an existing tool class: the tool stores a copy of the stored tool class instead of the one it was sent with. With custom.toolclass.validation: true, a tool with an unknown tool class is no longer stored (before, it was stored although the request failed with 400). See Tool classes.
  • Replacing a tool, version or tool class with an identical one succeeds (it failed with 500).
  • Replacing tools and versions increments their meta_version (it stayed at the initial value), as configured in custom.tool.meta_version.increment and custom.version.meta_version.increment. Adding, replacing or deleting a version also increments the meta_version of its tool, so clients see that the tool changed; set custom.tool.meta_version.on_version_change: false to keep the tool's meta version unchanged on version changes. See Meta versions.
  • Deleting the last version of a tool gives 400 Bad Request; delete the tool instead.
  • Versions without is_production are registered as non-production versions (they failed with 500).

Errors

Errors are problem details (title, status, detail), sent as application/json, instead of objects with message and code (see Errors). Requests that the validation of the API specification rejects get the reason as detail (e.g., 'FOO' is not one of ['CWL', ...]), no longer the generic The request is malformed.; set exceptions.validation_details: false to keep the generic one (see Errors).

Database

Without MONGO_DBNAME, the image uses the database trsStore (as configured) instead of trs-filer-db. Deployments with data in trs-filer-db (e.g., the Docker Compose setup of earlier versions) set MONGO_DBNAME=trs-filer-db, or move the data (see below); the Helm chart sets it already (value mongodb.dbName, by default trs-filer-db).

MongoDB 4.4 or later is required (Docker Compose and the Helm chart use 7.0). MongoDB 7.0 can't open data files of 3.6, which earlier Compose setups used: migrate them with mongodump (with MongoDB 3.6) and mongorestore (with 7.0), e.g., into trsStore, or upgrade one major version at a time. For the Helm chart, see Helm chart.

Image

  • Image tags: latest is the highest stable release from the first release with the new release process on, no longer the head of dev, which is tagged dev (and sha-<commit> per commit); dated tags are no longer published. Pin a release (X.Y.Z) in deployments (see Versioning and releases). Images published by the new workflows are scanned, attested and signed (see Verifying images and charts).
  • Gunicorn is configured by trs_filer/gunicorn_config.py and the environment variable GUNICORN_CMD_ARGS (e.g., --workers 2; see Server); GUNICORN_PROCESSES, GUNICORN_THREADS and TRS_CONFIG are no longer read, and Gunicorn always binds 0.0.0.0:8080.
  • Forwarded headers (X-Forwarded-*) are trusted from Gunicorn's default (127.0.0.1) only, no longer from every client: behind a proxy that terminates TLS, set FORWARDED_ALLOW_IPS to its addresses (the Helm chart sets it).
  • The image runs as user 1000 (also with a read-only root file system and a writable /tmp) and reports its version (e.g., in the service info) from the build argument SETUPTOOLS_SCM_PRETEND_VERSION.

Docker Compose

The service is named trs-filer (was trs), and the app is published at http://localhost:8080/ga4gh/trs/v2/ on the loopback interface only (was port 80 on all interfaces). MongoDB data lives in the named volume mongodb-data instead of ../data/trs/db: migrate it with mongodump and mongorestore if needed (see Database). The Compose setup applies the overlay config.dev.yaml (see Local overrides).

Helm chart

Releases attach the chart (see Verifying images and charts); upgrade with it, and your values file, as described in Deployment. Releases installed from the chart in the repository before (from the tag v0.3.0 or from dev) may need more steps, depending on two things, which you check before the upgrade:

  • The name of MongoDB's claim (kubectl get pvc -n <namespace>): the chart calls MongoDB's Deployment, Service and PersistentVolumeClaim mongodb-<appName> (trs_filer.appName, by default trs-filer); charts before July 2024, including that of v0.3.0, called them mongodb. Helm deletes resources that are no longer in the chart, so upgrading such a release deletes the claim mongodb and, with the reclaim policy Delete (the default of most storage classes), its volume and the data, and starts MongoDB on a new, empty claim.
  • The running MongoDB version: the chart runs MongoDB 7.0 (mongodb.image, see MongoDB). Charts before January 2026 ran mongo:3.6, whose data files 7.0 can't open; the chart on dev since then ran the floating mongo:noble (MongoDB 8.2, 8.3 or 9.0, depending on when the pod last started), whose data 7.0 can't open either (MongoDB doesn't support downgrades). Check the version (and the feature compatibility version) of the running server (with mongosh, or mongo, the old shell, in images before 6.0); OLD is the MongoDB Deployment (mongodb or mongodb-<appName>):

    kubectl exec deploy/OLD -n <namespace> -- sh -c \
      '"$(command -v mongosh || echo mongo)" --quiet --eval "printjson({version: db.version(), fcv: db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1}).featureCompatibilityVersion})"'
    

    If MongoDB doesn't start any more (CrashLoopBackOff, e.g., after a restart pulled a newer mongo:noble that can't open the data), its log names the versions (kubectl logs deploy/OLD -n <namespace> --previous): set an image that can open the data (e.g., kubectl set image deploy/OLD OLD=docker.io/library/mongo:8.3 -n <namespace>; the container is named after the Deployment), wait until it runs, then continue here.

Then:

  • Claim mongodb-<appName> and MongoDB 7.0: upgrade as usual.
  • Claim mongodb-<appName> and MongoDB 8.x or 9.0: set mongodb.image in the values file to the running major and minor version (e.g., docker.io/library/mongo:8.2, as db.version() reports it; mind the kernel requirements of 8.x, see MongoDB) and upgrade; or move the data to 7.0 by dump and restore as below.
  • Otherwise (claim mongodb, or MongoDB 3.6): dump the data before the upgrade and restore it afterwards, as below.

Dump and restore

The app is down from the first to the last step. The commands are for the release trs-filer with the default trs_filer.appName (trs-filer, the name of the app's Deployment and the label app of its pods; MongoDB's Deployment and claim are mongodb-trs-filer) and database (trs-filer-db, mongodb.dbName); replace them if yours differ. OLD is the MongoDB Deployment, and claim, before the upgrade (mongodb or mongodb-trs-filer). Clients can reach the app whenever it runs, also with the empty database between the upgrade (step 4, which starts the app again) and step 5, and the restore replaces what they write then: keep clients away during the migration (e.g., by blocking the host at a proxy in front of the cluster, or in a maintenance window).

  1. Stop the app and wait until its pod is gone, so that the data doesn't change during the dump:

    kubectl scale deploy/trs-filer -n <namespace> --replicas=0
    kubectl wait pod -l app=trs-filer -n <namespace> --for=delete --timeout=120s
    
  2. Count the documents of each collection, for a comparison after the restore, and dump the database to a local file; check that mongodump succeeds (the command prints Dump OK):

    kubectl exec deploy/OLD -n <namespace> -- sh -c \
      '"$(command -v mongosh || echo mongo)" --quiet trs-filer-db --eval "db.getCollectionNames().forEach(c => print(c, db[c].countDocuments ? db[c].countDocuments({}) : db[c].count()))"'
    kubectl exec deploy/OLD -n <namespace> -- \
      mongodump --quiet --db trs-filer-db --archive > trs-filer-db.archive \
    && echo "Dump OK"
    
  3. Keep the old volume as a fallback: set the reclaim policy of its PersistentVolume to Retain, which keeps the volume (with the data) when its claim is deleted:

    kubectl patch pv "$(kubectl get pvc OLD -n <namespace> \
      -o jsonpath='{.spec.volumeName}')" \
      -p '{"spec": {"persistentVolumeReclaimPolicy": "Retain"}}'
    
    • With the claim mongodb, also annotate the claim, so that Helm keeps it (Helm checks the annotation on the claim in the cluster before deleting it on upgrades): kubectl annotate pvc mongodb -n <namespace> helm.sh/resource-policy=keep.
    • With the claim mongodb-trs-filer (and MongoDB 3.6, or a version to leave), the upgrade keeps the claim, so MongoDB would start on the old data files; delete MongoDB's Deployment and the claim (the volume is retained), so that the upgrade creates them anew, with an empty volume: kubectl delete deploy/mongodb-trs-filer pvc/mongodb-trs-filer -n <namespace>.
  4. Update the values file (see Values) and upgrade the release with the chart of the release, e.g.:

    helm upgrade trs-filer trs-filer-0.4.0.tgz -n <namespace> -f values.yaml
    

    The chart of the tag v0.3.0 deploys the app with the API extensions/v1beta1, which Kubernetes no longer serves since 1.16; if helm upgrade fails because the release's manifest contains removed APIs, map them first with the mapkubeapis plugin (helm mapkubeapis trs-filer -n <namespace>).

  5. Stop the app again, so that it doesn't write during the restore, and wait until MongoDB has initialized the new volume (its entrypoint first runs a temporary server for the init script, then starts MongoDB) and accepts connections:

    kubectl scale deploy/trs-filer -n <namespace> --replicas=0
    kubectl wait pod -l app=trs-filer -n <namespace> --for=delete --timeout=120s
    timeout 300 sh -c 'until kubectl logs deploy/mongodb-trs-filer -n <namespace> \
      | grep -q "ready for start up"; do sleep 2; done' \
    && timeout 120 sh -c 'until kubectl exec deploy/mongodb-trs-filer -n <namespace> -- \
      mongosh --quiet --eval "db.runCommand({ping: 1}).ok"; do sleep 2; done' \
    && echo "MongoDB ready"
    

    Continue only if MongoDB ready is printed; otherwise check MongoDB's pod and log (kubectl get pods, kubectl logs): e.g., after a failed init, MongoDB restarts without it and doesn't print the line.

  6. Restore the dump, replacing the collections that the app has created, and count the documents again:

    kubectl exec -i deploy/mongodb-trs-filer -n <namespace> -- \
      mongorestore --quiet --drop --archive < trs-filer-db.archive \
    && kubectl exec deploy/mongodb-trs-filer -n <namespace> -- \
      mongosh --quiet trs-filer-db --eval 'db.getCollectionNames().forEach(c => print(c, db[c].countDocuments ? db[c].countDocuments({}) : db[c].count()))'
    

    If the restore fails on the options of an index, restore with --noIndexRestore: the app creates its indexes on startup. The restore is repeatable (--drop).

  7. Only if the restore succeeded and the counts match those of step 2, start the app:

    kubectl scale deploy/trs-filer -n <namespace> --replicas=1
    

Once the new release works, delete the old claim mongodb (if any: kubectl delete pvc mongodb -n <namespace>), the retained volume (kubectl delete pv <name>; this deletes only the PersistentVolume object: delete its storage, e.g., the disk, with your storage provider) and the local dump. The service_info collection of the dump is no longer used (see Service info).

Values

Since the chart of v0.3.0, the values changed:

  • clusterType is removed: the chart creates an Ingress (trs_filer.ingress.enabled, by default true; HTTPS via cert-manager with trs_filer.ingress.https) or, with the Ingress disabled, an OpenShift Route (with TLS at the router), as v0.3.0 did with clusterType: openshift. trs_filer.route.enabled (by default true) turns the Route off, e.g., for a Route of your own, and trs_filer.route.annotations sets its annotations (see Ingress and Route).
  • host_name has no default (it was a test host); set it.
  • trs_filer.image is elixircloud/trs-filer (released charts set the release's image), no longer lvarin/trs-filer:0.1.0.
  • trs_filer.resources and mongodb.resources set the requests and limits of the app and of MongoDB.
  • mongodb.image is docker.io/library/mongo:7.0 (was mongo:3.6, and mongo:noble on dev), pulled as mongodb.imagePullPolicy says; mongodb.dbName sets the app's database (trs-filer-db if empty).

The chart annotates MongoDB's claim with helm.sh/resource-policy: keep (see MongoDB), and replaces MongoDB's pod with the Recreate strategy. MongoDB's init script (ConfigMap mongo-init-script-<appName>, was mongo-init-script) creates the collections of the app's database only (no service_info, and nothing in trsStore); it runs on an empty volume only, so existing databases are not changed. The app gets the MongoDB host from MONGO_HOST (the Service mongodb-<appName>), also if the config Job fails. The chart in the repository has the placeholder version 0.0.0 (was 2.0.0); released charts have the release's version.

Images from dev since FOCA 0.17

Images from dev built since the move to FOCA 0.17 (dev, sha-<commit>) ran on an unreleased FOCA commit; releases run on FOCA 0.19. Besides the detail of rejected requests (see Errors), the startup warning about the placeholders of the service info changes: FOCA logs it (logger foca.api.service_info) while id, organization.url, contactUrl or documentationUrl are in a reserved example domain (RFC 2606, e.g., org.example.trs or https://trs.example.org), instead of the app (logger trs_filer.app), which warned of the placeholders id and organization (name or URL) of the base config only; a placeholder organization.name is no longer warned of. FOCA's plugin store (collection plugin_store) and plugin listing are for gateway apps: TRS-Filer creates no collections or indexes beyond its own, so its database user needs no new rights.