Configure a pipeline run

nf-core pipelines are configured at several levels: environment variables for Nextflow itself, parameters for the pipeline, and config files for execution. Using nf-core/demo, this tutorial explains each layer as worked examples.

By the end you will have:

• Supplied pipeline parameters on the command line and from a params.yaml file.
• Activated profiles to switch between container engines and test scenarios.
• Layered a custom custom.config file with process-specific overrides.
• Used ext.args (the convention for conf/modules.config) to extend a tool’s command line without editing the module.
• Used NXF_* environment variables to control Nextflow’s runtime.
• Understood which configuration source overrides which.

Baseline run

Before changing anything, run the pipeline with no custom configuration so you have something to compare against. This baseline run relies entirely on nf-core/demo’s built-in defaults. No environment variables, no custom parameters, and no extra config files are set. In each of the following steps you will replace one of these defaults and observe the effect.

Start with a plain test invocation as your reference point, replacing docker with your preferred software dependency manager (for example, singularity or conda).

nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results

In the background, this run uses three default configuration sources that you will override in the next steps:

• The pipeline’s internal nextflow.config and the test and docker profiles
• Default pipeline parameters (path to input samplesheet, the customisable MultiQC title, and others). See the nf-core/demo parameters page and nextflow_schema.json
• Your default Nextflow runtime settings (for example, work directory and Nextflow version)

Configure with parameters

Pipeline parameters are the knobs and switches a pipeline exposes to control how its steps run. For example, inputs, outputs, skip flags, reference data choices, and tool toggles. They’re documented per pipeline. For nf-core/demo, see the parameters reference and nextflow_schema.json.

You can supply them two ways:

• Directly on the command line
• From a params.yaml or params.json file.

The two forms are interchangeable, and you can mix them.

1. Set parameters on the command line. Any parameter is supplied as a --<name> flag:

   nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results_customcommandline --multiqc_title "nf-core/demo command line configured run"

   Output now goes to results_customcommandline/ instead of results/, and --multiqc_title sets the MultiQC report title.

2. For longer parameter sets, create a params.yaml file:

   outdir: results_customparamsfile
   multiqc_title: "nf-core/demo parameter file configured run"

3. Apply it with -params-file:

   nextflow run nf-core/demo -r 1.2.0 -profile test,docker -params-file params.yaml

   Output now goes to results_customparamsfile/ instead of results/. Open results_customparamsfile/multiqc/nf-coredemo-parameter-file-configured-run_multiqc_report.html in a web browser to confirm the report title was set.

Configure with config files

Config files are the most flexible configuration layer. They control both where the pipeline runs and how it runs. For example, which executor submits jobs (local, slurm, awsbatch), how much CPU and memory each process requests, what Nextflow does when a process fails, which container registry to pull from, where intermediate work lives, and how to apply settings to specific steps. They’re the right place for any setting that depends on your infrastructure rather than on what the pipeline does scientifically.

Nextflow assembles its final configuration from several config files. Nextflow picks up some config files automatically, some you switch on by name with -profile, and others you point it at explicitly with -c.

The sections that follow cover all three approaches and then show how to target specific processes inside any config file.

Auto-loaded nextflow.config files

Nextflow looks for config files in three places without being told.

1. The pipeline’s own nextflow.config in the project directory. Included with every nf-core pipeline’s source code. For example, nf-core/demo’s nextflow.config, which loads conf/base.config and declares every shipped profile. Don’t edit this file. See the warning in Configuration options.
2. $HOME/.nextflow/config. Your personal config, applied to every Nextflow run you launch. Good for settings that should follow you across all pipelines, such as your Singularity cache location or a default executor for your workstation.
3. nextflow.config in the launch directory. Applied automatically whenever you run Nextflow from that directory. Good for per-project or per-working-directory settings you don’t want to retype each time.

Add a nextflow.config in your current directory:

process {
  cpus = 1
  memory = 2.GB
}

Re-run the baseline command. Nextflow picks the file up automatically, without a -c flag:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results_customnextflowconfig

Compare in your web browser results/pipeline_info/execution_report_<datetimestamp>.html with results_customnextflowconfig/pipeline_info/execution_report_<datetimestamp>.html, looking at the table under the ‘Tasks’ section. Observe that the process NFCORE_DEMO:DEMO:COWPY has changed its memory from 4.GB to 2.GB, as specified in the new nextflow.config.

Activate bundled settings with profiles

A profile is a named bundle of configuration that lives inside a config file under the profiles scope. Use profiles to switch between configuration sets stored in a single config file. You activate one or more with the -profile flag. You’ve been using profiles since Baseline run. -profile test,docker activates two: test (a small public dataset) and docker (use Docker to manage software).

Every nf-core pipeline comes with a standard set of profiles:

• Software profiles: docker, singularity, apptainer, podman, conda, charliecloud, shifter (one per container engine or environment manager).

• Test profiles: test (a small dataset for quick verification) and test_full (the full-size dataset used in CI).

  Note

  The test_full profile in nf-core/demo is also a very small test dataset that can be used for testing. In other pipelines these can be much larger, but produce realistic output. Verify the size of tests of other pipelines before attempting to test_full on smaller machines such as laptops.

• Institutional profiles: contributed to nf-core/configs and loaded automatically by every nf-core pipeline. Activate one with -profile <institution> if your cluster has one. See Use shared institutional configs.

Combine profiles with commas. Order matters — later profiles override earlier ones where they overlap:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results

In this example, options in the docker profile override matching options in test.

You can also define your own profile for settings you’d like to reuse. Add a mymachine profile to the bottom of your existing launch-directory nextflow.config:

profiles {
  mymachine {
    process {
      cpus = 2
    }
  }
}

Activate it alongside the existing profiles:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker,mymachine --outdir results_customprofile

Compare the ‘Tasks’ section of results_customnextflowconfig/pipeline_info/execution_report_<datetimestamp>.html with results_customprofile/pipeline_info/execution_report_<datetimestamp>.html in your browser. Observe that the process NFCORE_DEMO:DEMO:COWPY has changed the CPUs from 1 to 2, as specified in the new nextflow.config.

Use shared institutional configs

If you work on a shared HPC cluster or cloud platform, there’s a good chance someone has already written a profile for it. The nf-core/configs repository collects over 150 cluster-specific configs contributed by the community. Each covers the executor, queue, resource limits, container engine, scratch paths, module systems, and other settings needed to run nf-core pipelines well in that environment.

Every nf-core pipeline loads these configs automatically — there’s nothing to download or copy yourself. At run time, each pipeline fetches the nfcore_custom.config from the nf-core/configs repository and makes every profile in it available alongside the pipeline’s own profiles. A handful of examples from the configs directory:

• uppmax — UPPMAX (Sweden)
• bih — Berlin Institute of Health
• aws_tower — Seqera Platform on AWS
• crick — Francis Crick Institute

Browse the full list at nf-co.re/configs.

You activate one the same way you activate any other profile. If your institution’s infrastructure is supported by nf-core/configs, replace <institution> with the profile name for your environment to try it out.

Some institutions also contribute pipeline-specific overrides — a profile that further tunes resources for a particular pipeline on a particular cluster. These live under conf/pipeline/<pipeline>/<institution>.config in the repository and are picked up automatically when you run that pipeline with the matching -profile.

Pass a config explicitly with -c

For configuration you don’t want loaded by default — a one-off resource bump, a shared institutional config, or an experimental override — pass it on the command line with -c.

1. Create a small file named custom.config:

   process {
     memory = 3.GB
   }

2. Apply it with -c:

   nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c custom.config --outdir results_customconfig

Compare the ‘Tasks’ section of results_customprofile/pipeline_info/execution_report_<datetimestamp>.html with results_customconfig/pipeline_info/execution_report_<datetimestamp>.html. Observe that the process NFCORE_DEMO:DEMO:COWPY has changed the memory from 2 to 3, as specified in the new custom.config. The CPUs also drop from 2 to 1. Without the mymachine profile, they fall back to the value in nextflow.config.

You can pass -c more than once. Nextflow applies the files in order. Later ones override earlier ones.

Tune how the pipeline executes

Beyond resource requests, config files control how each run executes. For example, which scheduler picks up jobs, what to do when a process fails, and where intermediate files live. These settings sit alongside the process scope you used above, with a few sibling top-level options.

Edit custom.config to add automatic retries and redirect the work directory:

workDir = 'nf-work'

process {
  memory = 3.GB
  cpus = 1
  errorStrategy = 'retry'
  maxRetries    = 2
}

Apply it the same way as before:

nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c custom.config --outdir results_customconfig2

Nextflow now creates the nf-work/ directory from the config in your launch directory, alongside the results directory.

Each setting controls a different aspect of the run. Common settings are:

• process.executor: where jobs go — local (default), slurm, awsbatch, lsf, pbs, kubernetes, and others.
• process.queue: which queue or partition to submit to on shared clusters.
• process.errorStrategy with process.maxRetries: retry transient failures (for example, a node going down or a timeout) instead of failing the whole run.
• workDir: where Nextflow stages intermediate files — point it at fast scratch storage on HPC to keep your home filesystem clean.

Target specific processes with withName and withLabel

The blanket process { memory = 3.GB } example above applies to every process in the pipeline. In practice you’ll usually want to target specific steps of the pipeline. For example, to bump the resources for an aligner without changing anything else, or pass an extra command-line argument to one tool. Nextflow gives you two selectors for this:

• withName matches a specific process by its fully qualified name. nf-core/demo runs four processes — FASTQC, SEQTK_TRIM, COWPY, and MULTIQC — and each has a name you can see in .nextflow.log or in pipeline_info/execution_trace.txt after a run.
• withLabel matches every process that carries a given label. All nf-core pipelines tag their processes with size labels (process_low, process_medium, process_high, process_high_memory) — see how they map to resource requests in nf-core/demo’s conf/base.config. A single withLabel block can adjust whole resource tiers at once.

Edit your custom.config to include both in the process block:

workDir = 'nf-work'

process {
  memory = 3.GB
  cpus = 1
  errorStrategy = 'retry'
  maxRetries    = 2

  withName: 'NFCORE_DEMO:DEMO:FASTQC' {
    cpus = 1
    memory = 3.GB
  }

  withLabel: 'process_low' {
    cpus = 1
    memory = 1.GB
  }
}

Re-run with the -c custom.config.

nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c custom.config --outdir results_customconfig3

Compare results_customconfig2/pipeline_info/execution_report_<datetimestamp>.html with results_customconfig3/pipeline_info/execution_report_<datetimestamp>.html. FASTQC should report requesting 1 CPU and 3 GB memory instead of 2 and 4 GB.

SEQTK_TRIM has also changed. It now requests 1 CPU and 1 GB instead of 2 CPUs and 4 GB. Process labels are declared on each process definition inside the module. For example, SEQTK_TRIM’s main.nf declares label 'process_low' on its second line.

Pass tool arguments with ext.args

ext.args is a per-process configuration value that nf-core modules use to inject extra command-line flags into the underlying tool. It lives inside a withName selector (so it follows the same targeting rules as the previous section), and the module picks it up at runtime with def args = task.ext.args ?: ''.

By convention, nf-core pipelines collect every per-process option in a dedicated file, conf/modules.config. It is another source of configuration loaded automatically through the pipeline’s nextflow.config. Open nf-core/demo’s conf/modules.config to see two real uses of ext.args:

withName: FASTQC {
    ext.args = '--quiet'
    // ...
}

withName: 'MULTIQC' {
    ext.args = { params.multiqc_title ? "--title \"$params.multiqc_title\"" : '' }
    // ...
}

FASTQC always runs with FastQC’s --quiet mode. MULTIQC dynamically picks up --title if the multiqc_title parameter is set. This is why the multiqc_title value from your params.yaml flowed through into the MultiQC report.

To pass your own flags without editing the module code, override ext.args in custom.config:

workDir = 'nf-work'

process {
  memory = 3.GB
  cpus = 1
  errorStrategy = 'retry'
  maxRetries    = 2

  withName: 'NFCORE_DEMO:DEMO:FASTQC' {
    memory = 3.GB
    cpus = 1
    ext.args = '--quiet  --nogroup'
  }

  withLabel: 'process_low' {
    memory = 1.GB
    cpus = 1
  }

  withName: 'MULTIQC' {
      ext.args = { params.multiqc_title ? "--title \"$params.multiqc_title\"" : '' }
  }
}

Re-run with the -c custom.config.

nextflow run nf-core/demo -r 1.2.0 -profile test,docker -c custom.config --outdir results_customconfig4

FASTQC picks up the new flags through task.ext.args in its main.nf. Compare results_customconfig3/fastqc/SAMPLE1_PE/SAMPLE1_PE_1_fastqc.html and results_customconfig4/fastqc/SAMPLE1_PE/SAMPLE1_PE_1_fastqc.html in your web browser. The ‘Per base sequence quality’ plot has changed because of the --nogroup option.

Sibling keys you’ll see in the same file:

• ext.args2, ext.args3 — second and third argument sets for modules that call more than one tool.
• ext.prefix — overrides the prefix used for the module’s output filenames.

For resourceLimits, executor tuning, and container registry overrides, see System requirements. For profile precedence and shared institutional configs, see Configuration options.

Configure with environment variables

NXF_* environment variables are the outermost configuration layer. Nextflow reads them from your shell as it starts, before it parses any config file or parameter. Use them for settings that don’t change between runs. For example, which Nextflow version to use, where the work directory and container cache live, and whether to operate offline. They configure Nextflow itself, not the pipeline. You cannot set pipeline inputs or outputs this way.

You set them like any other shell variable: with export, inline before a single command, or by adding them to your shell’s startup file.

1. Pin a Nextflow version and redirect the work directory for this run:

   export NXF_VER=25.10.4
   export NXF_WORK=$HOME/nf-demo-work
   nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results_customenv

   The version line at the top of stdout (and in .nextflow.log) should now report 25.10.4, and intermediate files should appear under $HOME/nf-demo-work/ instead of ./work/.

2. To set a variable for a single command without exporting it, prefix the invocation:

   NXF_VER=25.10.4 nextflow run nf-core/demo -r 1.2.0 -profile test,docker --outdir results_customenv2

3. To persist a variable across sessions, add the export to your shell config:

   export NXF_VER=25.10.4

Put it all together

Each layer you’ve explored solves a different problem. In practice, a single run usually uses several at once. This step combines them into one invocation so you can see how they interact.

1. Set environment variables for runtime concerns that don’t change between runs:

   export NXF_VER=25.10.4
   export NXF_WORK=$HOME/nf-demo-work

2. Capture pipeline parameters in a params-final.yaml file so the run is reproducible:

   outdir: my_results
   multiqc_title: "nf-core/demo configured run"

3. Put per-process overrides in custom.config:

   process {
     withName: 'NFCORE_DEMO:DEMO:FASTQC' {
       cpus = 4
       memory = 8.GB
       ext.args = '--quiet --nogroup'
     }
   
     withLabel: 'process_low' {
       cpus = 2
       memory = 4.GB
     }
   }

4. Launch the run, activating the test and docker profiles and passing the parameter and config files:

   nextflow run nf-core/demo -r 1.2.0 \
     -profile test,docker \
     -params-file params-final.yaml \
     -c custom.config

This single command exercises every layer the tutorial introduced. Each one is resolved independently:

• Runtime: NXF_VER and NXF_WORK are read from the shell before Nextflow parses any config, pinning the version and redirecting intermediate files.
• Configuration: config files layer in a fixed order — the pipeline’s bundled nextflow.config (which pulls in conf/base.config and conf/modules.config) loads first, the test and docker profiles override matching keys, and custom.config overrides them last. The withName and withLabel blocks in custom.config therefore win for FASTQC and every process_low step.
• Parameters: values in params.yaml set the pipeline parameters. A matching --flag on the command line would override them.

To confirm each layer took effect, check:

• my_results/pipeline_info/execution_report_<datetimestamp>.html — the withName block requests 4 CPUs and 8 GB for FASTQC. On this small test dataset, the test profile’s resourceLimits = [cpus: 2, memory: '4.GB'] caps the request, and the report shows 2 CPUs and 4 GB. Drop the test profile (or raise its resourceLimits) to see the full 4 CPUs and 8 GB.
• my_results/multiqc/nf-coredemo-configured-run_multiqc_report.html — the report title should read “nf-core/demo configured run”. (Setting multiqc_title also renames the report file.)
• $HOME/nf-demo-work/ — intermediate files should appear here instead of under ./work/.

Next steps

• Learn more about configuration options (profiles, shared nf-core/configs, and precedence rules)
• Learn more about system requirements (resource limits, executors, and tool argument overrides)
• Learn more about Nextflow configuration