Create a queue

A Queue is a processing queue managed by AWS Batch or GCP Cloud Batch. The workflow orchestrator — Nextflow or Cromwell — submits jobs to it.

You can register several queues to keep work organised and to manage cost. Queues also configure the instance type used.

AWS and GCP differ here

AWS queues carry more settings: initial disk size, instance families, and whether to use spot or on-demand instances. GCP queues are simpler, because Cloud Batch derives the machine from each task’s declared CPU, memory and disk requirements. See Cloud resources.

Create a spot queue

export ENVIRONMENT_ID=<environment-id>

geni queue create \
  --name "SPOT" \
  --environment-id $ENVIRONMENT_ID \
  --execution-mode spot

Create an on-demand queue

Create a second queue in on-demand mode. Analysts then pick per submission, and can name it as the fallback for spot-capacity failures — see Managing submissions. It can also be targeted directly by specific Nextflow processes — see Routing tasks to a different queue.

geni queue create \
  --name "ONDEMAND" \
  --environment-id $ENVIRONMENT_ID \
  --execution-mode ondemand

Verify

geni queue list
geni engine list

With an environment, an engine and at least one queue in place, the project is ready. An analyst can now register a workflow and submit — see Managing workflows.

Disable a queue

Disable a queue to retire it without deleting its cloud resources. A disabled queue keeps its infrastructure, history and costs, and you can enable it again at any time.

While it is disabled:

• Blocked: new submissions and retries that target it, including as a fallback queue, for every role, administrators included. They fail with 409 Queue is disabled.
• Not blocked: submissions already running, which finish normally; updating or deleting the queue; logs and cost reports.

geni queue disable <queue-id>   # asks for confirmation; --force skips it
geni queue enable <queue-id>
geni queue list --disabled

In the web console, administrators have Disable and Enable buttons on the queue’s page. From Python:

client.queue_disable("<queue-id>")

Disabling and enabling require an administrator account.

Troubleshooting

If an AWS queue ends up FAILED, run geni queue revisions <id> and geni queue logs <id> to see the underlying CloudFormation error — see Troubleshoot a failed deployment. GCP queues hold no cloud resources, so logs always reports unavailable for them; a GCP queue failure means the queue record itself (its config) is wrong, not a deployment.