Development¶
See Contributing for how to contribute; this page describes the development setup, the checks, the documentation and the versions.
Development needs Python 3.13 or later and
uv (e.g., pipx install uv; see its
installation guide).
To set up a development environment with the versions of the lock
(uv.lock, see Dependencies), as CI does, and run the
checks that CI runs:
uv sync --locked --extra dev # the app, with the dev dependencies, in `.venv`
source .venv/bin/activate # or `activate.fish` etc.
pre-commit install # run the checks on every commit
pre-commit run --all-files # ruff, mypy, semgrep, ast-grep, actionlint
pre-commit run --all-files --hook-stage manual gitleaks-all # secrets
coverage run -m pytest # the tests, with coverage
coverage report
Instead of activating .venv, commands can be run with uv run, which
syncs the environment first, so give it the extra (e.g.,
uv run --locked --extra dev pytest). pip install -e ".[dev]" also works, but installs the
newest versions in the ranges of pyproject.toml instead of the locked
ones, so its results can differ from CI's.
The tests (tests/, mirroring the package) need no database: app tests
create the app from the base config (trs_filer/config.yaml) and the test
overlay (tests/config_overlay.yaml, with the settings each deployment must
make, for a fictitious identity provider that the tests mock), and send
requests through Flask's test client, with access tokens that the tests sign
(make_headers() in tests/conftest.py), and with an in-memory database
(mongomock); they have no network
access (see tests/conftest.py). CI also runs them against MongoDB 7.0; to
do so locally, set MONGODB_TEST_URI to a single-node MongoDB without other
data, one per test run (the tests drop the database trsStore before each
test, and refuse to start if it exists), e.g.:
docker run -d --rm --name mongodb-test -p 127.0.0.1:27017:27017 mongo:7.0
until docker exec mongodb-test mongosh --quiet --eval 'db.adminCommand({ping: 1}).ok'; do sleep 1; done
MONGODB_TEST_URI=mongodb://localhost:27017 pytest
docker stop mongodb-test
The tests of object storage (tests/test_storage.py) use an in-memory S3
(moto). CI also runs them against
SeaweedFS (with MongoDB, step "Start S3-compatible server" of
.github/workflows/ci.yml, whose image is the version of the Docker Compose
example in File storage; bump both by
hand); to do so locally, set S3_TEST_ENDPOINT_URL to an S3-compatible
server without other data (the tests create and delete the bucket
trs-filer-test), and, if it checks credentials, S3_TEST_ACCESS_KEY_ID
and S3_TEST_SECRET_ACCESS_KEY (test by default), e.g.:
docker run -d --rm --name s3-test -p 127.0.0.1:8333:8333 chrislusf/seaweedfs:4.51 server -s3 -ip.bind=0.0.0.0
until curl -s -o /dev/null http://127.0.0.1:8333; do sleep 1; done
S3_TEST_ENDPOINT_URL=http://127.0.0.1:8333 pytest tests/test_storage.py
docker stop s3-test
Coverage is measured for trs_filer, scripts and the docs hooks
(docs/hooks.py), and must stay at 100%: coverage report and
coverage xml fail below it (fail_under in pyproject.toml), and so do
Codecov's checks of the whole code and of each pull request's changes
(codecov.yml; CI reports coverage to Codecov,
see the badge in the README). New code, also new scripts, needs tests that
cover it completely; add tests that fail without your change, and exclude
unreachable code only with a reason (# pragma: no cover). CI also runs
the app with Docker Compose and sends requests to it (job "Smoke test" of
.github/workflows/ci.yml: public reads, writes refused without a token,
then writes with authentication disabled, see Running without
authentication); this test runs in CI
only, with no local equivalent other than docker compose up --build
--wait and requests by hand.
To try changes in the running app, rebuild it with
docker compose up --build -d. python -m trs_filer.app runs the app with
Flask's development server instead of Gunicorn (see
Server); it needs a MongoDB at db.host of the
config (e.g., set MONGO_HOST=localhost for a MongoDB published on the
host).
Run pre-commit in the environment with the dev dependencies: the mypy and
ast-grep hooks use the installed packages. The first run sets up the other
hooks, which takes a few minutes.
The checks are configured in
.pre-commit-config.yaml
and
pyproject.toml.
Project-specific rules are in .semgrep/ (security) and .ast-grep/
(anti-patterns such as HTTP status codes as strings, print() calls or
calls of the root logger), each with tests.
actionlint checks the workflows in
.github/workflows/, and their shell scripts with
ShellCheck if it is installed (as on
GitHub-hosted runners, so CI may find more than a local run without it). To
suppress a false positive, add a comment with the reason to the line:
# noqa: <code> (ruff), # type: ignore[<code>] (mypy),
# nosemgrep: <rule-id>, # ast-grep-ignore: <rule-id> or
# gitleaks:allow.
Running without authentication¶
Writes need an access token (see Access control),
also with Docker Compose, whose development overlay (config.dev.yaml) trusts a fictitious identity provider. To set
your own, see Local overrides. To try
writes locally without an identity provider, disable authentication with the
overlay
config.noauth.yaml,
which sets FOCA's api.specs.trs.disable_auth (false in the base
config), after config.dev.yaml, via TRS_FILER_CONFIG_OVERLAY (which
docker-compose.yaml passes to the app as FOCA_CONFIG_OVERLAY):
TRS_FILER_CONFIG_OVERLAY=/etc/trs-filer/config.dev.yaml:/etc/trs-filer/config.noauth.yaml \
docker compose up --build --wait
All operations are then public, writes too, and FOCA warns on startup that
authentication is disabled (Authentication is disabled for spec 'trs').
Tools then get no owners, and the roles on tools can't be managed
(/tools/{id}/permissions answers 403 Forbidden).
Run docker compose up --wait without the variable to enable it again.
Only use this on your machine (Docker Compose publishes the app on the
loopback interface only), never for deployments.
Dependencies¶
The dependencies are declared, with version ranges, in pyproject.toml
(which installs of the package, e.g., with pip, use) and locked in
uv.lock, with exact versions and hashes, by
uv (also a dev dependency, so that the lock
pins the version of uv that CI uses). CI's lint, test and docs jobs install
the locked versions, checked against their hashes (uv sync --locked, with
uv of a pinned astral-sh/setup-uv, at the version of the lock), so that
Dependabot's pull requests, which change only the lock, are tested on the
versions they lock; the lint job also checks that the lock matches
pyproject.toml (uv lock --check). The app itself is installed editable
there, built with build requirements from the package index (not the lock),
as pip install -e does; those jobs hold no publishing credentials. The
build requirements of the app (build-system.requires) are locked the same
way, as the dependency group build-system: CI's push job derives the
version of dev images with them, exported by the uv image of the
Dockerfile (pinned by digest) and installed with hash checks (by the pip
of the runner's Python, ensurepip, nothing downloaded), so that no
workflow step of a job that holds publishing credentials installs a package
without hash checks (tests/test_workflows.py checks this, and that the
group and build-system.requires are the same: keep them so). Two uv
versions are in use: that of uv.lock (astral-sh/setup-uv, updated with
the uv entry of Dependabot) and that of the Dockerfile stage (docker
entry).
The image build, which these jobs run too, installs the locked
runtime dependencies only (uv export --frozen --no-dev
--no-emit-project, which leaves out the extras and the group
build-system, then uv pip install --system --require-hashes
--only-binary :all: --no-deps, see the Dockerfile), so two builds of the
same commit get the same packages, and a new release of a dependency gets
into the image only through a reviewed change of the lock. The image
build builds the app's wheel with the locked build requirements (group
build-system),
installed with hash checks, too, and without network, so it downloads
nothing that isn't hash-checked; it fails unless the base image has the
locked FOCA release (scripts/image_requirements.py), so update the
foca requirement, the lock and the base image together, by hand
(Dependabot updates neither FOCA nor its image). The image
contains no pip and no uv: the FOCA base image has none (since FOCA 0.20),
and uv is mounted from the Dockerfile stage pinned by digest into the
steps that use it, which the smoke test of CI checks (step "Check that
image has no installer"; tests/test_image_build.py checks the
Dockerfile). After adding, removing or changing a dependency in
pyproject.toml, update the lock and commit both:
uv lock # update `uv.lock` after changes to `pyproject.toml`
uv lock --upgrade-package <name> # upgrade one locked package
uv lock --check # what CI checks: the lock matches `pyproject.toml`
Dependabot updates pyproject.toml and uv.lock together, including the
build-system group (uv entry of .github/dependabot.yml), and the uv
image of the Dockerfile (docker entry).
Helm chart¶
The Helm chart is in deployment/, a thin chart over
FOCA's library chart foca-common (see Helm library
chart
in FOCA's docs). deployment/Chart.yaml pins its version and
deployment/Chart.lock locks it; helm dependency build deployment
downloads it into deployment/charts/ (not committed), and
deployment/charts.sha256 has its digest, which CI and the release
workflow check, as OCI tags can be pushed again. Check changes with
Helm, e.g.:
helm dependency build deployment
(cd deployment && sha256sum -c charts.sha256)
helm lint deployment --strict -f deployment/ci/test-values.yaml
helm template trs-filer deployment -f deployment/ci/test-values.yaml
To update the library chart, change its version in deployment/Chart.yaml
(and the $comment and the copied parts of deployment/values.schema.json,
which follows the library's app-values.schema.json), then run helm
dependency update deployment and replace the digest in
deployment/charts.sha256 with that of the new package (cd deployment &&
sha256sum charts/foca-common-<version>.tgz > charts.sha256). The chart
renders the library's templates with the app's settings
(trs-filer.settings in deployment/templates/_helpers.tpl), which also
checks for values of earlier charts (trs-filer.checkLegacyValues) and,
on upgrades, for releases of earlier charts (trs-filer.checkUpgrade).
CI's job "Chart" (.github/workflows/ci.yml) lints the chart as the
release workflow does, with the values that users must pass (e.g., the
name of MongoDB's Secret and a fictitious identity provider):
helm lint deployment --strict -f deployment/ci/test-values.yaml. It
also runs the tests that render it (tests/test_chart.py; skipped locally
without Helm, failing without the library chart in deployment/charts/):
CI's job "Chart install" installs the chart in a cluster: on
kind, with the image built from the commit
(loaded into the cluster's node, so that nothing is pushed), the values of
deployment/ci/test-values.yaml and a Secret with MongoDB's credentials. It
waits for the rollout, then checks the API through a port forward: the
service info, public reads, and that writes need a token (401, as the
test values trust a fictitious identity provider). It then upgrades the
release with authentication disabled (config.api.specs.trs.disable_auth,
see Running without authentication; for
this test only) and checks the registration and retrieval of a tool, whose
URL must have the host name of the values (hostName). To do the same
locally, with Docker, kind, kubectl and Helm (in Bash):
kind create cluster --name trs-filer
docker build --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=0.0.0+kind \
--tag trs-filer:ci .
kind load docker-image trs-filer:ci --name trs-filer
kubectl create namespace trs-filer
kubectl create secret generic trs-filer-mongodb-credentials -n trs-filer \
--from-literal=root-username=root \
--from-literal=root-password="$(openssl rand -hex 24)" \
--from-literal=username=trs-filer \
--from-literal=password="$(openssl rand -hex 24)"
helm dependency build deployment
helm install trs-filer deployment -n trs-filer \
-f deployment/ci/test-values.yaml \
--set trsFiler.image=trs-filer:ci \
--set trsFiler.imagePullPolicy=Never \
--wait --timeout 600s
kubectl port-forward -n trs-filer service/trs-filer 8080:8080 &
until curl --fail http://localhost:8080/ga4gh/trs/v2/service-info; do
sleep 2 # the port forward may still be starting
done
kind delete cluster --name trs-filer # tear down
trsFiler.imagePullPolicy=Never makes the cluster use the loaded image
(with the chart's default, Always, it would pull it from a registry); the
image names its tag, so trsFiler.tag stays empty. Send further requests
as in the job's step "Check API" (.github/workflows/ci.yml).
Documentation¶
This documentation is built with MkDocs and
Material for MkDocs, from
docs/ and
mkdocs.yml,
by Read the Docs (configured in
.readthedocs.yaml)
and in CI; warnings, e.g., of broken links, fail the build. The API
reference is generated from the docstrings with
mkdocstrings. To build it locally, or serve
it with live reloading, in the dev environment set up as above, with the
locked docs dependencies too (as CI does; uv sync removes the extras that
it isn't given, so dev stays in the command; Read the Docs installs the
docs extra with pip, from the ranges of pyproject.toml):
uv sync --locked --extra dev --extra docs
mkdocs build --strict # into `site/`, ignored by Git
mkdocs serve
Parts of the docs are kept elsewhere, so that they are written only once: the
introduction, the known deployments and the quick start of the home and
getting started pages are sections of README.md (between
--8<-- [start:<name>] and --8<-- [end:<name>] comments), and the
deployment page is the README of the Helm chart
(deployment/README.md), which docs/hooks.py includes, with its links to
pages of the docs and to files of the repository adapted (tested in
tests/test_docs_hooks.py). Links in the chart's README must be absolute (to
https://trs-filer.readthedocs.io/en/latest/ and to files on branch dev on
GitHub), as they must also work in the packaged chart.
Versioning and releases¶
The project adopts the semantic versioning scheme for versioning. Currently the service is in beta stage, so the API may change without further notice.
The version is stored in Git tags (vX.Y.Z, pre-releases vX.Y.ZrcN):
the package version is derived from them (via
setuptools-scm), and the service
info reports it. Builds without Git metadata, such as local Docker builds
(e.g., with Docker Compose), report 0.0.0.dev0; the workflows pass the
version to the image build (build argument SETUPTOOLS_SCM_PRETEND_VERSION).
Images on Docker Hub (elixircloud/trs-filer) are tagged:
X.Y.Z: releaseX.Y.Z;latest: the highest stable release (not an alpha, beta or release candidate) onmain;dev: the latest commit ondev;sha-<commit>: commit<commit>ondev(full SHA), to deploy a fixed commit. Only commits that are still the head ofdevwhen their CI run reaches the image job get images.
The current release version is taken from one source, so that a release
doesn't need to bump it in many places: the release tag and, on dev, the
version in CITATION.cff (set before tagging, see below; from then on, the
latest docs show the upcoming version, and before the first release, they
show X.Y.Z). The docs and the chart's README (deployment/README.md) write
it as a placeholder, version in double curly braces with a space inside
each pair, and the chart version (e.g., 1.2.3-rc1 for 1.2.3rc1, as in the
names of the packaged charts) as chart_version in the same braces
(VARIABLE and CHART_VARIABLE in scripts/chart_readme.py; this page
can't show them, as they would be replaced), also in commands and links. The
docs hooks (docs/hooks.py) replace them with the version of CITATION.cff,
and scripts/chart_readme.py with the release version when the chart is
packaged; files read on GitHub (e.g., deployment/README.md) show the
placeholders, and sections of README.md included as snippets can't use
them. Comments, templates and tests use placeholders (e.g., 1.2.3) and
refer to sections by anchor. tests/test_version_mentions.py fails if the
current release version is mentioned as a release of the app (as a tag,
image or chart, or as "release" or "version" in prose; not, e.g., as the
version of a dependency) anywhere else than in CITATION.cff,
docs/releases/, the tests of the release tooling, links to the notes and
assets of a release, and migration headings (e.g., "From versions before
X"), which are about that release itself.
Features and fixes are merged into dev. Branch main only ever moves
forward to commits on dev that are released. First, shortly before the
release, set its version and date in the citation metadata, which Zenodo
archives with each release, and merge the change into dev; each version,
also each pre-release (e.g., X.Y.Zrc1), needs its own such change:
scripts/citation.py uses the standard library only and is tested in
tests/test_citation.py; check (below) fails unless version is X.Y.Z
and date-released is a date that is not in the future (UTC).
Then, to release version X.Y.Z from commit <sha> on dev (with that
change), after CI passed for it:
git switch --detach <sha>
# The release workflow's checks that need no tag, as it runs them
python scripts/release_version.py outputs vX.Y.Z
python scripts/citation.py check X.Y.Z
python scripts/check_release_pins.py
! grep -n TODO docs/releases/X.Y.Z.md # curated notes, if any
[ ! -f docs/releases/X.Y.Z.md ] || python scripts/release_notes.py check docs/releases/X.Y.Z.md
readme="$(mktemp -d)/README.md" && cp deployment/README.md "${readme}"
python scripts/chart_readme.py "${readme}" vX.Y.Z # links of the chart
helm lint deployment --strict -f deployment/ci/test-values.yaml # as `prepare`
helm package deployment --version X.Y.Z --app-version X.Y.Z -d "$(mktemp -d)"
git push origin <sha>:main # fails unless it is a fast-forward
git tag -s vX.Y.Z <sha> -m "vX.Y.Z"
git push origin vX.Y.Z
Tags can't be moved or deleted (see the repository settings below), and the
release workflow checks the tagged commit, so a check that fails after the
tag is pushed fails again on re-runs: the release then needs a new patch
version, with its own citation metadata and notes. Run these checks before
tagging, therefore. The CI job "Release readiness" runs the pin and the
notes checks, too, for the version in CITATION.cff while it has no tag (or
its tag is on the checked commit): on pull requests and in forks (which
often lack upstream's later tags) as warnings, on pushes to dev as errors,
which fail the CI run that the release workflow requires for the release
commit. The image scan of the release workflow can still fail after the
tag, e.g., on a vulnerability published since CI last scanned the image;
the remedy is the same.
The tag must be annotated and signed (git tag -s), with a key that GitHub can
verify (added to the tagger's GitHub account). The release
workflow,
triggered by the tag, checks that GitHub verified the signature, that the
signed tag object has the name of the tag, that the
tag's commit is on main and its CITATION.cff has the release version and
a date that is not in the future (UTC), that the dependencies are releases
(see below), and that the CI
workflow
passed for it (a run for the push of the commit to dev; if CI hasn't
finished, failed or was cancelled, e.g., the image job of a commit that a
later one superseded, re-run CI, then the release workflow). It packages
the Helm chart with the release version as version and appVersion, so
that it deploys the release image (elixircloud/trs-filer:X.Y.Z; the
repository's appVersion is dev), with the library chart (checked
against deployment/charts.sha256). In the packaged README,
links to the latest docs and to files on branch dev point to the release
(/en/vX.Y.Z/, blob/vX.Y.Z/); the release fails if any link to a version
that moves is left (scripts/chart_readme.py). The chart in the repository
is unchanged. After approval in the release environment, it builds the
image, pushes it by digest (untagged, with SBOM and provenance attestations)
and checks that the app reports the release version; it then scans exactly
that digest for vulnerabilities, in a job without credentials. After a
second approval in the release environment (once the scan passed), it
checks that the chart it downloads is the one it packaged (a single file
with the name and SHA-256 recorded when packaging; if not, the job fails
before it attests, signs or tags anything), attests the build
provenance of the scanned digest and signs it (see
Verifying images and charts),
tags exactly that image X.Y.Z (and latest, if it is the highest stable
release on main), attests and signs the chart, and creates a GitHub
release with the chart (.tgz), its Sigstore bundle (.tgz.sigstore.json)
and the release notes (see below). Releases are immutable (see the repository
settings below): once published, their assets can't be added, replaced or
deleted. The workflow therefore creates the release as a draft, uploads the
assets and publishes it last. A re-run finishes a draft left by an earlier
attempt; if the release is already published (e.g., by hand), it uploads
nothing and fails unless the release has exactly the assets that the run
built (only the Sigstore bundles may differ, as every attempt signs anew).
The error says what to do: a new patch version, or, if releases are
not immutable, deleting the release and re-running. Add curated notes to the
changelog in mkdocs.yml (nav; the docs build fails
otherwise), and refer to pages of the docs by name rather than by relative
links, which don't work in GitHub releases. No packages are published to
PyPI. Tags of alpha, beta and release candidate versions (e.g., vX.Y.Zrc1,
also aN and bN) get chart version X.Y.Z-rc1 (or -aN, -bN) and never
update the latest image. These and, until 1.0.0, all other releases are
GitHub pre-releases (never marked as the latest release). Digests of runs
that fail before publish (also in the scan) stay on Docker Hub, untagged
and unsigned; if publish fails after signing, the digest stays signed
(and possibly tagged), and a re-run of the failed jobs finishes it.
Re-running all jobs of the release workflow builds, scans, signs and tags a
new image digest (the image build is not reproducible); an earlier digest
that was published stays signed. Re-running only the failed jobs reuses the
digest already pushed (and scans it again if the scan failed). Re-running
all jobs also packages the chart again and replaces the chart artifact; the
release checks the new chart against its new SHA-256. Images of
commits on dev are published by the CI workflow (environment
dev-images): it pushes the image by digest, scans it, and attests, signs
and tags exactly that digest once all other jobs passed.
The CI workflow builds the image of every pull request and scans it with
Trivy (the shared action
scan-image
of the organisation, pinned to the commit of a release), and a weekly
workflow (scan.yml) scans latest. For pushes to dev and for releases,
the workflows push the image by digest, untagged, scan exactly that digest
in a job without credentials or signing identity, and only if the scan
passes attest, sign and tag it, so that the published images are the
scanned ones. The scans fail on critical and high vulnerabilities that have
a fix, and upload the findings that have a fix (all severities) to code
scanning (Security tab; only in this repository, not for pull requests from
forks, nor in the release workflow); findings without a fix appear there
once a fix is released. The weekly scan also uploads a full report,
findings without a fix included, as the workflow artifact
trivy-report-release-image of each run (trivy-report.json and
trivy-report.txt, a table; Actions tab, workflow "Scan"); it is not
uploaded to code scanning. Accepted findings are left out of all results.
To accept a finding (e.g., if it doesn't affect the app), add it to
.trivyignore.yaml with the affected package versions (purls), the
reason (statement) and an expiry date (expired_at), after which the
scans fail again. Trivy also scans for secrets (critical and high ones fail
the scans too); accept those under secrets: in the same file. Dependabot
updates the actions, including the pins of the shared actions
(@<commit> # vX.Y.Z, all on the same release, see
tests/test_workflows.py), to new releases of elixir-cloud-aai/.github;
the Trivy version is the default of the shared action's trivy-version
input, which Dependabot doesn't bump here: change it in the shared action,
whose release then reaches this repository as a Dependabot update of the
pins. Signing needs no secrets beyond the Docker Hub ones: it uses the
workflows' OIDC tokens (permission id-token: write).
The release workflow derives the version, the chart version, the kind of
release (stable, GitHub pre-release), whether the image is tagged latest
and the previous release (for the generated notes) from the tag with
scripts/release_version.py, which is tested over a table of tags
(tests/test_release_version.py) and checked like the app (ruff, mypy,
coverage); change these rules there, not in the workflow. It finds the
GitHub release of the tag, if any, and checks the assets of a published one
with scripts/release_assets.py (tested in tests/test_release_assets.py).
To test unreleased changes of FOCA, pyproject.toml may pin a FOCA commit
(foca @ git+https://github.com/elixir-cloud-aai/foca@<sha>), with FOCA's
development image of that commit as the base image (tagged by date, e.g.,
elixircloud/foca:YYYYMMDD-py3.13, pinned by digest, as the tag moves with
later commits of that day). The Dockerfile records the commit as
## FOCA commit: <sha>, which the tests check (tests/test_config.py); the
image build doesn't reinstall FOCA, but fails unless the base image has FOCA
of that commit (scripts/image_requirements.py). Such pins are temporary:
the release workflow fails while a requirement (also an optional one, a
dependency group or a build requirement) is a direct reference or names a
pre-release or a local version (e.g., +g<commit>), or a FOCA image has no
release tag (scripts/check_release_pins.py, tested in
tests/test_check_release_pins.py), so replace them by the released
versions before tagging. This applies to pre-releases of the app, too: they
are built on FOCA releases.
Release notes follow the format agreed for all repositories of ELIXIR On-Cloud:
- An intro paragraph saying what the release brings, with a link to the upgrade steps (e.g., on the Upgrading page) if there are any.
- Grouped sections, with empty ones left out, in this order:
## Breaking changes,## Features,## Security,## Fixes,## Documentation,## Maintenance. They hold bullets only, and each bullet ends with its issue and pull request numbers, e.g.,(#12, #34)(several pairs separated by semicolons, e.g.,(#12, #34; #56, #78)). ## Commits: GitHub's generated notes, verbatim (including "New Contributors"), without their## What's Changedheader, as they credit the authors.- Last, GitHub's generated
**Full Changelog**: ...line (no compare link of our own).
The curated notes in docs/releases/X.Y.Z.md (for pre-releases, e.g.,
X.Y.Zrc1.md) hold parts 1 and 2, without a title (the release's title is
its tag); the release workflow adds parts 3 and 4 from GitHub's
generate-notes
API, for the changes since the previous published release in the history of
the release commit (for stable releases the previous stable release, or the
previous pre-release if there is no stable one, as the generated notes can't
start before the first tag; for alpha, beta and release candidate versions
the previous release of any kind). For the first release (no published
release before it, even if older tags exist, such as v0.3.0), the API
would start at an older tag of its choice, so the workflow lists all merged
pull requests in the history of the release commit itself, in the API's
format: scripts/release_notes.py first reads the closed pull requests
(gh api --paginate --slurp
"repos/<owner>/<repo>/pulls?state=closed&per_page=100", for which the job
has pull-requests: read) and the commits (git rev-list HEAD at the
release commit); run these commands locally to preview the notes. Generated
notes in an unexpected format (without the **Full Changelog** line) are
kept as they are, with a warning, as the image is published by then.
Without curated notes, the release gets GitHub's generated notes alone.
Curated notes are published as written, so the release workflow fails if
they contain TODO, are not in this format or link to versions that move
(the latest docs, /en/latest/, or files on branch dev, blob/dev/; link
the release's, e.g., /en/vX.Y.Z/, blob/vX.Y.Z/, as the published notes
don't change) or have relative links (see below)
(scripts/release_notes.py check, tested in
tests/test_release_notes.py, which also checks the notes in
docs/releases/), and so does the CI job "Release readiness" on pushes (see
above); on re-runs, it replaces the notes of an existing release (also
overwriting manual edits).
Relative links, e.g., ../upgrading.md#from-040, work in the docs built
from docs/releases/, but not in the release body: link the release's docs
with their full URL, e.g.,
https://trs-filer.readthedocs.io/en/vX.Y.Z/upgrading/#from-040,
or within the page with #.... The check reports every link target without
a scheme (such as https: or mailto:) of inline links, images, reference
definitions and the HTML attributes href, src and srcset (each of its
URLs), anywhere in the notes, code blocks included (and lines of paragraphs that
start like a definition, [label]: ...). Detailed upgrade steps belong on the
Upgrading page, linked from the intro.
Maintainers: the workflows rely on these repository settings:
- environment
dev-images, limited to branchdev; - environment
release, with a required reviewer, "Prevent self-review" and a deployment policy that allows only tagsv*; - the Docker Hub secrets (
DOCKERHUB_LOGIN,DOCKERHUB_TOKEN) only in the two environments, not as repository secrets; - a tag ruleset for
v*that restricts who may create them and prevents moving or deleting them; - immutable releases;
mainprotected against non-fast-forward pushes;- a ruleset for
devthat requires the checkCI passed(the gate of the CI workflow) only.
While CITATION.cff names a version that has no tag yet, findings of
"Release readiness" (e.g., a pin to an unreleased commit) also stop the
dev image from being published; fix them or push the tag.
The Docker Hub organization is the repository variable DOCKERHUB_ORG (by
default elixircloud; forks set it to the organization they publish to,
with the Docker Hub secrets in an environment dev-images), so that the
scan jobs, which have no secrets, know the image name too, and the released
chart's default image (trsFiler.image) is the published one. Forks
without DOCKERHUB_ORG publish nothing: their CI runs the same checks (the
scan of a local build included, without the upload to code scanning) and
passes without the secrets, and their weekly scan is skipped.
On Read the Docs, the default version is stable (the highest release)
from the first release on, and the default branch is left empty, so that
latest follows the repository's default branch, dev; an automation
rule activates the versions of tags (v*), so that the docs linked from
packaged charts exist.
The release workflow and CI's jobs "Chart" and "Chart install" install a
fixed Helm version (version: of azure/setup-helm in release.yml and
ci.yml), and "Chart install" a fixed kind version (version: of
helm/kind-action). Dependabot updates the actions, but not these
versions: bump them by hand, Helm in all three places.