Create an engine

An Engine configures which workflow orchestrator is used — Nextflow or Cromwell. That choice determines the supported workflow language: Nextflow or WDL respectively.

Engines do three more things:

• Limit concurrency. An engine also acts as a queue, capping how many submissions run in parallel.
• Scope bucket access. An engine restricts which S3 or GCS buckets the workflows it runs may touch — read only, write only, or read and write.
• Hold scratch space. Each engine has its own bucket for temporary files, with a configurable lifetime for those files. Incomplete multipart uploads to that bucket are aborted after 1 day, so an interrupted transfer never lingers as billed storage.

Cost can be aggregated by engine as well as by queue. Creating engines and queues costs nothing, in cloud charges or in GENI Workflows licensing — so create as many as your work actually distinguishes between.

Create

export ENVIRONMENT_ID=<environment-id>

geni engine create \
  --name "Nextflow" \
  --environment-id $ENVIRONMENT_ID \
  --engine-version 26.04.6-2 \
  --read-write <bucket> \
  --read-only <bucket> \
  --write-only <bucket>

Argument	Meaning
<environment-id>	The ID GENI generated, for example 320e43e5-b171-4d0b-ada3-7cfcc71f6dd8
<bucket>	A bucket the engine’s workflows may access — an S3 URI for an AWS environment, a GCS URI for a GCP one, for example gs://geni-test-bucket

--read-write, --read-only and --write-only are repeatable, and each accepts a prefix as well as a whole bucket.

Note

--engine-version defaults to 26.04.6-2, a single Nextflow image that runs on both AWS and GCP. It supports Nextflow config profiles, and its retries reuse finished tasks even after a cancelled run — see Managing submissions. No version is pinned by provider; any registered version is accepted on either cloud.

Caution

The engine needs read access to every bucket holding input files and write access to the bucket a submission writes its output to. A missing grant surfaces as a permission error part-way into a run, not at submission time.

Disable an engine

Disable an engine to retire it without deleting its cloud resources. A disabled engine keeps its infrastructure, history and costs, and you can enable it again at any time.

While it is disabled:

• Blocked: new submissions and retries that target it, for every role, administrators included. They fail with 409 Engine is disabled.
• Not blocked: submissions already running, which finish normally; updating or deleting the engine; logs and cost reports.

geni engine disable <engine-id>   # asks for confirmation; --force skips it
geni engine enable <engine-id>
geni engine list --disabled

In the web console, administrators have Disable and Enable buttons on the engine’s page. From Python:

client.engine_disable("<engine-id>")

Disabling and enabling require an administrator account.

Troubleshooting

If the engine ends up FAILED, run geni engine revisions <id> and geni engine logs <id> to see the underlying deployment error — see Troubleshoot a failed deployment.

Next

Update an engine.