Managing workflows

Once you are signed in, the first step is to register a workflow and its first version, or a new version of an existing workflow.

List workflows

geni workflow list

See the versions of one workflow:

geni workflow versions <workflow-name>

Every listing command accepts --format, which takes table (the default), csv or json.

Register a workflow or version

geni workflow create \
  --workflow-name <workflow-name> \
  --workflow-version <workflow-version> \
  --file <path to the Nextflow, WDL or ZIP file>

Recommendations

Naming

Use a workflow name without spaces, so it is easy to pass around. The GitHub repository name is a good choice.

Use semantic versioning for the version (v1.2.3). Because every workflow and version is available to every environment, a suffix is a useful way to flag work in progress — v1.2.3-dev.

The Nextflow (.nf), WDL (.wdl) or ZIP file must be on your computer. The CLI uploads it to GENI, which keeps it for future submissions. Once registered, you no longer need the local file.

ZIP layout

Nextflow. The archive must contain a main.nf at its root. Every dependency must be inside the archive. Configuration files (nextflow.config) are allowed. Python scripts belong in a bin/ folder at the root — Nextflow then copies them, as executables, into the container of every task in the workflow.

WDL / Cromwell. The archive must contain a main.wdl at its root, with every dependency inside it.

zip -r <workflow-files>.zip main.nf nextflow.config modules/

Time limits

Use --cancel-time to set the maximum duration of a submission of this workflow version. After that period GENI cancels the submission automatically.

geni workflow create \
  --workflow-name wgs-germline \
  --workflow-version v2.1.0 \
  --file wgs-germline.zip \
  --cancel-time 00:12:00

The format is DD:HH:MM. There is no limit by default.

Caution

Configure --cancel-time, especially for workflows under development. Without it, a pipeline that hangs keeps burning compute until someone notices.

A submission can override this with its own --cancel-time.

Immutability

By default, uploading a file with an existing workflow name and version replaces that version’s file. Mark a version immutable to block that: GENI then rejects any upload with the same name and version.

Immutability only protects against overwrites. An immutable version can still be deleted and restored, and it has no effect on submissions that are running or already finished.

Set it when you register the version:

geni workflow create \
  --workflow-name wgs-germline --workflow-version v2.1.0 \
  --file wgs-germline.zip --immutable true

An administrator can change it later, in either direction:

geni workflow update <workflow-name> --workflow-version <workflow-version> --immutable true
geni workflow update <workflow-name> --workflow-version <workflow-version> --immutable false

In the web console, administrators get a switch in the Immutable column of the versions table on the workflow’s page. Turning it off asks for confirmation first. From Python:

client.workflow_version_set_immutable("wgs-germline", "v2.1.0", True)

A deleted version must be restored before its immutability can be changed.

Tags

Tags attached to a workflow are inherited by every submission of every version. See Tagging.

geni workflow create \
  --workflow-name wgs-germline --workflow-version v2.1.0 \
  --file wgs-germline.zip --tag client=acme --tag project=dnabr

Export a workflow

geni workflow export <workflow-name> \
  --workflow-version <workflow-version> \
  --output-folder <path>

Delete

Delete one version:

geni workflow delete \
  --workflow-name <workflow-name> \
  --workflow-version <workflow-version>

Deleting a workflow with no --workflow-version would delete every one of its versions, so it fails unless you also pass --force:

geni workflow delete --workflow-name <workflow-name> --force

In every case the command asks for confirmation before deleting, unless --force is passed — --force is required to delete a whole workflow, and as a side effect it also skips the prompt.

Delete is a soft delete: the version’s file and its parent workflow record are kept, not removed from storage, so it can always be brought back — see Restore. An immutable version can be deleted too; immutability only blocks overwrites.

Restore

Deleting a workflow or version doesn’t destroy it — it can be restored later. Restoring requires an administrator account; analysts can’t restore.

Restore one version:

geni workflow restore \
  --workflow-name <workflow-name> \
  --workflow-version <workflow-version>

Restore a workflow and all of its versions:

geni workflow restore --workflow-name <workflow-name>

Restoring a version also restores its parent workflow, if that was deleted too — see User roles.

Administrators can also restore from the web console:

• A version. On the workflow’s page, tick Show deleted versions, then use the restore button on the deleted row.
• A whole workflow. On the Workflows page, tick Show deleted workflows, then use the restore button on the deleted row.

From Python, use client.workflow_version_restore(name, version) or client.workflow_restore(name).

Next

Managing submissions — running the workflow you just registered.