I/O Schemas¶
Experimental
Support for I/O Schemas follows a draft of the GA4GH I/O Schema, the profile version 0.3.0, which may still change; so may this support, also in minor releases of TRS-Filer. The check is off by default.
The GA4GH I/O Schema describes the inputs and outputs of a workflow as a JSON Schema, with keywords for forms (titles, widgets, files) and for the workflow engine (which tool version and descriptor it belongs to, and the names and types of the engine's parameters). Portals generate forms from it and turn what users enter into the parameters of a run (e.g., via WES).
An I/O Schema is published in TRS as a file of the tool version, as the TRS convention of the profile defines; the TRS specification is not changed. TRS-Filer stores and serves such files like any other file, with or without the check.
Publishing an I/O Schema¶
For each descriptor type of a tool version, register the I/O Schema as a file with:
- the file type
OTHER(tool_file.file_type); - the path
ga4gh-io.schema.jsonin the directory of the primary descriptor (tool_file.path, e.g.,ga4gh-io.schema.jsonfor the primary descriptorhello.wdl, orwdl/ga4gh-io.schema.jsonforwdl/main.wdl); - the descriptor type that it describes (
type, e.g.,WDL), the one that itsx-ga4gh-workflow.descriptor_typenames; - its content (
file_wrapper.content), or a URL that TRS-Filer retrieves it from (file_wrapper.url, see Remote files).
The schema names the meta-schema of the profile in $schema, and the tool
version and the checksum of its primary descriptor in x-ga4gh-workflow, so
that clients can check that it belongs to the version they fetched it for.
Choose the identifiers of the tool and the version (PUT), so that the
schema can name them; generated identifiers are not known before the
registration.
For example, the I/O Schema of a WDL workflow hello.wdl
(hello.wdl), which greets someone once or
several times:
{
"$schema": "https://raw.githubusercontent.com/uniqueg/ga4gh-io-schema/v0.3.0/schemas/ga4gh-io-profile.schema.json",
"$id": "https://example.org/schemas/hello/wdl/1.0.0",
"title": "Hello",
"description": "Greets someone, once or several times.",
"x-ga4gh-workflow": {
"trs_tool_id": "HELLO",
"trs_version_id": "1.0.0",
"descriptor_type": "WDL",
"descriptor_type_version": "1.1",
"primary_descriptor_path": "hello.wdl",
"checksum": [
{
"type": "sha-256",
"checksum": "d9fd1db6e20b9373a4ccecf4ea40145eb00b9fea36d111b6248c8696e28310d4"
}
],
"entry_point": "hello"
},
"type": "object",
"properties": {
"inputs": {
"type": "object",
"properties": {
"name": {
"type": "string",
"minLength": 1,
"title": "Name",
"description": "Who to greet.",
"x-ga4gh-parameter": {
"type": "String"
}
},
"times": {
"type": "integer",
"minimum": 1,
"maximum": 10,
"default": 1,
"title": "Times",
"description": "How often to greet.",
"x-ga4gh-ui-widget": "slider",
"x-ga4gh-parameter": {
"type": "Int"
}
}
},
"required": ["name"],
"additionalProperties": false
},
"outputs": {
"type": "object",
"properties": {
"greeting": {
"type": "string",
"title": "Greeting",
"x-ga4gh-file": {
"kind": "file",
"mediaType": "text/plain"
},
"x-ga4gh-parameter": {
"type": "File"
}
}
}
}
}
}
tool.json registers the workflow with its
I/O Schema as version 1.0.0 of the tool HELLO (with an access token of
an admin, see Access control):
curl -X PUT "http://localhost:8080/ga4gh/trs/v2/tools/HELLO" \
-H "Authorization: Bearer <token>" \
-H "Content-Type: application/json" \
-d @docs/examples/io-schema/tool.json
Fetching an I/O Schema¶
Clients find the I/O Schema of a tool version among its files, and fetch it by its path:
- List the files of the tool version for the descriptor type:
GET /tools/HELLO/versions/1.0.0/WDL/fileslists, e.g.,{"path": "ga4gh-io.schema.json", "file_type": "OTHER"}next to the primary descriptor. - Fetch it:
GET /tools/HELLO/versions/1.0.0/WDL/descriptor/ga4gh-io.schema.jsonreturns it in aFileWrapper, ascontent, with itschecksum;PLAIN_WDLin place ofWDLreturns it as is (see Descriptor types). - Check it as the profile describes: against the meta-schema, and its
x-ga4gh-workflowagainst the tool version and the checksum of the primary descriptor.
Checking I/O Schemas¶
With custom.io_schema.validation (false by default), TRS-Filer checks
I/O Schema files when tools and versions are registered, and rejects
invalid ones with 400 Bad Request:
I/O Schema files are files of type OTHER named ga4gh-io.schema.json, in
any directory; other files aren't checked. A file is invalid if it:
- is not a JSON object;
- names another meta-schema than that of profile version 0.3.0 in
$schema(https://raw.githubusercontent.com/uniqueg/ga4gh-io-schema/v0.3.0/schemas/ga4gh-io-profile.schema.json), e.g., of another version of the profile; - doesn't validate against that meta-schema (which also checks that UI
widgets fit the types of their schemas, and that the parameters directly
under
inputsandoutputsuse only widgets for inputs and outputs, respectively; nested schemas and definitions aren't checked for this; see the profile's changelog); or - names another descriptor type than its own in
x-ga4gh-workflow.descriptor_type.
The detail of the response says why, and, for schemas that don't validate,
where (as a JSON path) and which keyword of the meta-schema they break, but
not their values (names that could be values, e.g., with :, / or @,
are replaced by *), e.g.:
{
"title": "Bad request",
"status": 400,
"detail": "The I/O Schema file (ga4gh-io.schema.json) doesn't validate against the meta-schema of the GA4GH I/O Profile 0.3.0 at $.properties (keyword 'required': 'inputs')."
}
The validator of the
profile
reports all errors with their values; check schemas with it before
registering them. Files retrieved from their URLs are checked once
retrieved; the check applies to every registration of tools and versions
(POST and PUT).
Validating against the meta-schema takes time that grows about linearly
with the size of a schema, also for deeply nested ones. Measured on a
developer laptop, the example and about a hundred inputs in ten groups take
well below a second, a thousand inputs about half a second; larger files
take about 10 to 25 s per MiB, depending on their annotations (e.g., UI
widgets). So the validation runs in a process of its own, which is stopped
after custom.io_schema.timeout seconds (default: 10, including a
fraction of a second to start the process and load the meta-schema); files
not checked within it are rejected, too (400 Bad Request). The default
can thus reject valid files from a few hundred KiB, well below the default
request size limit of 1 MiB. Raise the limit (e.g., to 30) if you expect
larger schemas, keeping in mind that registrations take as long. The limit applies to each file; the app checks
one file at a time (per Gunicorn worker), and waiting for another check
counts against the limit. The process takes about 50 to 100 MB of memory
and about 0.2 s of CPU time to start, so give the app enough of
both when enabling the check (e.g., resources of the Helm chart, whose
default CPU limit of 100m makes that about two seconds). The app
must be installed (as in the image), as the process imports it.
The meta-schema of profile version 0.3.0 and its vocabularies are vendored
unchanged in
trs_filer/io_schema/,
from commit d44e8d4 of the profile's v0.3 branch (Apache License 2.0);
nothing is fetched. To vendor another version, replace the directory (named
after the date and commit of the source) and update PROFILE_VERSION and
META_SCHEMA_DIR in trs_filer/io_schema/__init__.py, the checksums in
tests/test_io_schema.py, the example (docs/examples/io-schema/, whose
tool.json holds the other two files) and this page.
TRS-Filer doesn't check the identifiers and the checksum of
x-ga4gh-workflow against the tool version (identifiers may be generated
on registration). Tool versions stored before the check was enabled are not
checked, nor those registered with replicas of the app that don't check yet
while the option is rolled out.
In the Helm chart, set the option in config (see the chart's
Configuration):