Nextflow config profiles

Nextflow’s -profile flag selects one or more named blocks from a profiles { } section in nextflow.config, letting a single workflow support several configurations — a small test run and a full production run, for instance — without maintaining a second copy of the pipeline. GENI exposes it as --nextflow-profile on geni submission create.

geni submission create \
  --workflow-name wgs-germline \
  --workflow-version v2.1.0 \
  --params-file sample.json \
  --engine-id <engine-id> \
  --queue-id <queue-id> \
  --nextflow-profile test \
  --output-folder s3://lab-results/run-4471/

Pass more than one profile as a comma-separated list — --nextflow-profile test,docker. Nextflow merges them left to right, so a later profile’s settings win where they overlap.

Requirements

Nextflow only. --nextflow-profile is rejected with 422 for a Cromwell workflow.

Engine version 26.04.6-1 or later. Older Nextflow engine images — 25.10.4, 26.04.1-s3, 26.04.1-gcp — predate this feature and reject the flag with 422 before a submission is even created. See Update an engine to move an existing engine onto 26.04.6-1.

The workflow must be registered as a ZIP, with nextflow.config at the root alongside main.nf. A bare .nf file has no config to select a profile from. See Registering a workflow for how GENI extracts a ZIP.

Declare the profiles

Nothing GENI-specific — this is the profiles { } block Nextflow already supports, in the workflow’s own nextflow.config:

profiles {
  test {
    params.samplesheet = 's3://lab-data/test/samples.csv'
    process {
      withName: 'GATK_HAPLOTYPECALLER' {
        cpus = 2
        memory = '8 GB'
      }
    }
  }

  production {
    params.samplesheet = 's3://lab-data/production/samples.csv'
    process {
      withName: 'GATK_HAPLOTYPECALLER' {
        cpus = 16
        memory = '64 GB'
      }
    }
  }
}

What a profile can and can’t change

A profile can set params.* and per-process resources (cpus, memory, container, and so on) — exactly what the example above does.

A profile cannot redirect execution off the queue or engine GENI resolved for the submission. geni.config — the file GENI generates from the engine and queue you passed — is loaded as the last -c file, and Nextflow config is merged path-by-path with later files winning on the same path. So whatever geni.config sets for process.executor, process.queue (AWS), or the google.* keys (GCP) always wins over the same path in your profile, no matter which profile is selected. This is the same mechanism that lets GENI’s generic process.queue coexist with a workflow’s own withName:/withLabel: selectors — see Routing tasks to a different queue for the full precedence rules.

In practice: use profiles for workflow parameters and process-level resources, and route specific processes to a different queue or use the fallback queue for anything about where a task runs.

Retries reuse the same profile

geni submission retry re-reads the submission’s stored settings, including the profile it was created with — you don’t pass --nextflow-profile again:

geni submission retry <submission-id>

The retried run’s log shows the same -profile flag as the original submission, alongside -resume if cache reuse is enabled.

Verify which profile ran

geni submission logs <submission-id>

The head log’s Running: nextflow run ... line shows the exact -profile argument (or its absence, for a submission with none), and Nextflow itself prints workflow.profile if your pipeline logs it.