Hi everyone, I would like to write tests for a genomics pipeline using nf-test, so -since I'm new to both nextflow and nf-test- I read the manual and created my first single-module and chained-module tests. Everything works as expected! ...so now I'd like to tweak the test procedure :) The pipeline is composed of

samtools

and

paraphase

, and runs

paraphase

with the

-g

flag followed by a comma-separated list of 10 genes. During testing, would it be possible to break this list in 10 different pipeline invocations, each with 1 gene? This would help keeping each test small and fast to execute. I started with gene GBA. I followed this example, and in my

tests/nextflow.config

file I wrote:

process {
    withName: PARAPHASE {
        ext.args = '-g GBA'
    }
}

This setup works as intended: the config instructs

paraphase

to run against GBA only and produces the output that I expect. However, I cannot figure out how to write the remaining 9 blocks for the remaining 9 genes: any new section in the config will have the same

withName: PARAPHASE

, so besides the string containing the gene's name, all the 10 configurations will look identical. I looked around for a way to make them distinct, so that I can run

paraphase

with 10 different cli invocations, but I cannot figure out how. What am I missing? Have you ever encountered this scenario?

Is there an easy way to programatically create one of the "subway" flowcharts or does one have to mess around in a photo editor? We have a fork of an existing workflow and we added a new pathway and new tools. We want to document that in flowchart form.

Hi. The documentation states that templates are discouraged (https://www.nextflow.io/docs/latest/process.html#template) "*Template* scripts are generally discouraged due to the caveats described above. The best practice for using a custom script is to embed it in the process definition at first and move it to a separate file with its own command line interface once the code matures." Is this correct? If so, when I create a command line script, does Nextflow still pass args to it, or do I have to pass it expliclitly? When using template, the script section looks like

script:
    template 'gprofiler2_gost.R'

However, if this were a command line, would I need to write it as

RScript gprofiler_gost.R $args

or just

RScript gprofiler_gost.R

and args will be passed automatically? Thank you

Hello, I am trying to run nfcore/funcscan on a HPC and keep hitting the following error:

executor >  slurm (14)
[55/710edc] NFC…(ERZ23877681-contig.fa.gz) | 4 of 4, cached: 4 ✔
[30/7657be] NFC…NNOTATION:PYRODIGAL (N073) | 4 of 4, cached: 4 ✔
[92/06c6af] NFC…AA (N073_pyrodigal.faa.gz) | 4 of 4, cached: 4 ✔
[33/0bc528] NFC…NA (N073_pyrodigal.fna.gz) | 4 of 4, cached: 4 ✔
[54/e97e11] NFC…BK (N073_pyrodigal.gbk.gz) | 4 of 4, cached: 4 ✔
[84/be13af] NFC…AMP:AMPLIFY_PREDICT (N072) | 0 of 4, retries: 4
[4c/54bf06] NFC…:AMP:MACREL_CONTIGS (N072) | 2 of 4, cached: 2
[7a/3ed0e0] NFC…N073.macrel.prediction.gz) | 2 of 2, cached: 2
[2b/9d3e82] NFC…73.macrel.all_orfs.faa.gz) | 2 of 2, cached: 2
[98/ce1802] NFC…:FUNCSCAN:AMP:AMPIR (N075) | 0 of 4
[9d/97345b] NFC…:AMP:AMP_DATABASE_DOWNLOAD | 1 of 1, cached: 1 ✔
[-        ] NFC…:AMP:AMPCOMBI2_PARSETABLES -
[-        ] NFC…CAN:AMP:AMPCOMBI2_COMPLETE -
[-        ] NFC…SCAN:AMP:AMPCOMBI2_CLUSTER -
[-        ] NFC…_FUNCSCAN:FUNCSCAN:MULTIQC -
ERROR ~ Error executing process > 'NFCORE_FUNCSCAN:FUNCSCAN:AMP:AMPLIFY_PREDICT (N075)'

Caused by:
  Process `NFCORE_FUNCSCAN:FUNCSCAN:AMP:AMPLIFY_PREDICT (N075)` terminated with an error exit status (137)


Command executed:

  AMPlify \
       \
       \
      -s 'N075_pyrodigal.faa'

  #rename output, because tool includes date and time in name
  mv *.tsv N075.amplify.tsv

  cat <<-END_VERSIONS > versions.yml
  "NFCORE_FUNCSCAN:FUNCSCAN:AMP:AMPLIFY_PREDICT":
      AMPlify: $(AMPlify --help | grep 'AMPlify v' | sed -e "s/^.*AMPlify v//")
  END_VERSIONS

Command exit status:
  137

Command output:
  (empty)

Command error:
  INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_TASK_WORKDIR is set, but APPTAINERENV_NXF_TASK_WORKDIR is preferred
  INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
  2025-08-03 22:30:18.853526: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
  /usr/local/bin/AMPlify: line 2:    45 Killed                  /usr/local/share/amplify/src/AMPlify.py $@

Work dir:
  /mnt/shared/scratch/neckersl/private/funcscan/work/02/e2658c3294d6e97e0034a06ad1ce01

Container:
  /mnt/shared/scratch/neckersl/private/funcscan/work/singularity/depot.galaxyproject.org-singularity-amplify-2.0.0--py36hdfd78af_1.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details
ERROR ~ Pipeline failed. Please refer to troubleshooting docs: <https://nf-co.re/docs/usage/troubleshooting>

 -- Check '.nextflow.log' file for details

I have tried increasing the memory of this task, I now have it at 250gb but it is still erroring which makes me think its something else. Any suggestions would be greatly appreciated. Thanks.

Try asking this in the #C02K5GX2W93 channel

I didn't realise there was one. Wasn't showing on my list of channels. Found it now, thanks.

Hello everyone! 🙂 Hopefully this is the right channel, I've run into a somewhat complicated issue whilst try to customise the #C045UNCS5R9 pipeline. In essence, I'm trying to process microarray .CEL files without external access to bioconductor, there's a module called ``differentialabundance/modules/nf-core/affy/justrma` . The main.nf for this looks like so:

process AFFY_JUSTRMA {
    tag "$meta.id"
    label 'process_single'

    conda "${moduleDir}/environment.yml"
    container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        '<https://depot.galaxyproject.org/singularity/bioconductor-affy:1.78.0--r43ha9d7317_1>':
        'biocontainers/bioconductor-affy:1.78.0--r43ha9d7317_1' }"

    input:
    tuple val(meta), path(samplesheet), path(celfiles_dir)
    tuple val(meta2), path(description)

    output:
    tuple val(meta), path("*.rds")             , emit: rds
    tuple val(meta), path("*matrix.tsv")       , emit: expression
    tuple val(meta), path("*.annotation.tsv")  , emit: annotation, optional: true
    path "versions.yml"                        , emit: versions

    when:
    task.ext.when == null || task.ext.when

    script:
    template 'affy_justrma.R'
}

Importantly, within the R script, there's a function that's used to conditionally fetch files from bioconductor packages like so:

install_cdf_db <- function(celfile, annotation = FALSE){
    library(affyio)
    headdetails <- read.celfile.header(celfile)
    ref.cdfName <- headdetails[[1]]
    cleaned.cdfName <- cleancdfname(ref.cdfName, addcdf = FALSE)

    exts = 'cdf'
    if (annotation){
        exts <- c(exts, '.db')
    }
    options(timeout=600)
    for (package in paste0(cleaned.cdfName, exts)){
        install.packages(
            package,
            lib = 'libs',
            repos = BiocManager::repositories(),
            dependencies = c("Depends", "Imports")
        )
    }
    cleaned.cdfName
}

importantly

repos = BiocManager::repositories()

points towards a URL which R can use to download conditionally based on the cel files (I've been using the

test_affy

profile, where package resolves to

hgu133plus2cdf

. This repo of course has to follow the structure of bioconductor repository. What I've managed to do so far is get this working locally - That is, create a local directory called

mirror

which contains all the files required for the

test_affy

profile and get the module to use this repository, I mount it into the container using:

containerOptions { params.affy_mirror_dir ? "-v ${params.affy_mirror_dir}:/mirror" : '' }

in the module main.nf - which allows me to provide a path the local mirror of the repository, I've made the following modification to the R template.

options(timeout=600, BIOCONDUCTOR_CONFIG_FILE ='$affy_config_yml', BIOCONDUCTOR_ONLINE_VERSION_DIAGNOSIS = FALSE)
for (package in paste0(cleaned.cdfName, exts)){
        install.packages(
            package,
            lib = 'libs',
            repos = c(BioC_mirror = "file:///mirror/bioc_mirror"), // hardcoded path within the container where the mirror will be mounted
            dependencies = c("Depends", "Imports")

However, the issue is, is that eventually the hope is store a full mirror of bioconductor on s3. However, I can't seem to figure out pass this mirror directory when stored on s3 to the Rscript to use to install bioconductor packages. My intuition would be to try and load the s3 version as a parameter (e.g. make a param s3_mirror_dir which could be`s3://dev-data/mirror/`) pass that into a channel -

Channel.fromPath('params.s3_mirror_dir')

and then somehow make the R script aware from it within the container, but am aware that

containerOptions

won't support an s3 path to mount directly in. Very happy to add to/clarify anything I've missed, and any help is massively appreciated. 🙂

Hello, I hope you are doing well. I was trying to make TxDb object from GFF an d I Had warning like that and the txdb object couldn't be created, is there any solution? Thank you

Hi. Not sure if this is the right place to ask, but I'm using VS Code and have some problems with the Nextflow extension auto-formatting. 1. Harshil alignment seems to be ignored

tuple val(meta), path("*_filtered*.tsv") , emit: filtered
    path "versions.yml"                      , emit: versions

turns into

tuple val(meta), path("*_filtered*.tsv"), emit: filtered
    path "versions.yml", emit: versions

when I format the document 2. There are some other weird code changes, like

container "${ workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container ?
        '<https://depot.galaxyproject.org/singularity/pandas:1.5.2>' :
        'biocontainers/pandas:1.5.2' }"

turning into

container "${workflow.containerEngine == 'singularity' && !task.ext.singularity_pull_docker_container
        ? '<https://depot.galaxyproject.org/singularity/pandas:1.5.2>'
        : 'biocontainers/pandas:1.5.2'}"

Do I need to place something in settings.json, like ??

"[nextflow]": {
        "editor.defaultFormatter": "nextflow.nextflow"
    }

Thank you

@Merlin Szymanski and I are currently having an issue with github’s raw links in the test datasets repository. Viewing raw links in other repos works fine, but for https://github.com/nf-core/test-datasets, we both get a 404 error. Is something wrong with the test-datasets repo? Do others have the same issue?

I am trying to run the nf-core/metatdenovo pipeline on the ifb cluster and it keeps crushing. If I post the error could someone help me?

Hello, I have a dumb question. Where can I found which container are used in a nf-core pipeline and which version is used for traceability? I am currently trying to use a "simple nf-core pipeline", bamtofastq. Thank you very much for your help !

Has anyone successfully integrated an nf-core pipeline into a larger, custom, nextflow pipeline? The goal is to able to treat several nf-core pipeline as ‘workflows’ as part of a much larger neoantigen prediction pipeline im building. It requires a lot of steps and existing nf-core pipelines (sarek, rnaseq, hlatyping, epitopeprediction) cover the majority of the steps. I’ve gotten it to work manually but it could be done automatically as everything is connected

I've ran into the issue where certain channels are dropped after

resume

after a specific

join

operation in

nf-core/scnanoseq

(https://github.com/nf-core/scnanoseq/blob/05d705a301a262669c2252c890106e31a28a120e/subworkflows/local/quantify_scrna_isoquant.nf#L95C16-L95C17) I was reading some of the discussions and also found the gotcha explaining some of it here: https://midnighter.github.io/nextflow-gotchas/gotchas/join-on-map-fails-resume/ I was now simply wondering what the best way is to implement this is (nf-core) workflows. If it's using the proposed solution in that gotcha, I'm a little bit lost on how to properly include that in a pipeline :') Edit: maybe slightly different issue (?)

has anyone face this error in the CI/CD:

Installed distributions
Creating settings.xml with server-id: github
Overwriting existing file /home/runner/.m2/settings.xml
Run ./setup-nextflow/subaction
  with:
    version: 24.10.5
    all: false
  env:
    NXF_ANSI_LOG: false
    NXF_SINGULARITY_CACHEDIR: /home/runner/work/quantms/quantms/.singularity
    NXF_SINGULARITY_LIBRARYDIR: /home/runner/work/quantms/quantms/.singularity
    CAPSULE_LOG: none
    TEST_PROFILE: test_dda_id
    EXEC_PROFILE: docker
    JAVA_HOME: /opt/hostedtoolcache/Java_Zulu_jdk/17.0.16-8/x64
    JAVA_HOME_17_X64: /opt/hostedtoolcache/Java_Zulu_jdk/17.0.16-8/x64
Input version '24.10.5' resolved to Nextflow undefined
Error: Cannot read properties of undefined (reading 'includes')
Error: Could not run 'nextflow help'. Error: Unable to locate executable file: nextflow. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.

Hi, have a bit of a naive question. If we decide that we'd rather not run one of the tools currently set to run, cancel the pipeline and resume it without said tool, does it work or does the entire pipeline restart?

hello i am new here and i try to use nf-core scrna starsolo to ran my scrna-seq data. because it is 150bp pair end sequencing, so the barcodelength is not 28 (default), but 150, what is the best practice to use customized parameters

--soloUMIfiltering -  --soloMultiMappers EM --soloCBstart 1 --soloCBlen 16  --soloUMIstart 17 --soloUMIlen 12\

N E X T F L O W  ~ version 25.04.2
nextflow run nf-core/scrnaseq -r 2.0.0

let me know if more info is required. Thank you!

Hi everyone, dumb question: can someone explain the impact of the

executor.perJobMemLimit

vs

executor.perTaskReserve

settings with the

lsf

executor? What would happen when setting one (or both) to true and vice versa ? Greatly appreciate it 🙏

Is there anyone here with python scripting experience with pandas/numpy/scipy/matplotlib/scipy who would be willing to look at a likely 'tiny' papercut bug issue in mag? https://github.com/nf-core/mag/issues/383 TL;DR: fixing a python script to handle if you have a distance matrix of just one object (so there is no distance)?

Hello everyone, I hope you are all doing well. I have never used Nextflow before (every time I look through the website, I get a little overwhelmed, haha), and I have some scripts in R/Python that I want to link up into a nice pipeline. Can someone suggest a good tutorial for doing so? Thank you for your time :D

I am trying to download a pipeline and associated containers for the rnavar pipeline, using `nf-core pipeline download`` . However, I am using a mac for this setup, and Singularity/Apptainer is required to pull the containers:

ERROR    Singularity/Apptainer is needed to pull images, but it is not installed or not in $PATH

Setting up

Singularity/Apptainer

on a mac is... more work than I expected. Is there another way to pull the containers outside of the nf-core tools system?

A question to the Python professionals here (which I am not, obviously): I read a number of posts regarding the use of custom (unpublished) Python projects inside a nextflow pipeline. My problem seems to have a different root, though. My pipeline runs with docker and I try to use officially released docker images throughout. One of the tools, however, sets some parameters in a

params.py

module which is part of the tool code base inside the docker image. I tried adding a modified

params.py

to the

/bin

directory in the pipeline and adding this path to the

PYTHONPATH

variable. I checked that

PYTHONPATH

is correctly set and that

params.py

is accessible inside the nextflow module. However, it does not seem to work, and the module (and the parameters) are still loaded from the original file. Any ideas why that is? Any hints how to deal with tools that hard-code parameters (except building custom docker images)?

A naive question, I have created a minimum dataset to test the #C084Z8NLZFB pipeline at

<https://github.com/fmobegi/nf-core-test-datasets/tree/abotyper>

. The guiding notes (https://nf-co.re/docs/tutorials/adding_a_pipeline/test_data) suggest creating a new branch at https://github.com/nf-core/test-datasets/branches/active and then make a pull request to use that branch as the target. For some reason, I am unable to create this branch. Any pointers will be appreciated.

I noticed that the results for denovotranscript pipeline on the website are incomplete. The pipeline failed during the Trinity module because "Essential container in task exited". I'm guessing it has to do with spot instances. is it possible to launch a full test manually with different settings to get around this? Slack conversation

Quick question regarding nextflow configs. I thought that if I specify multiple configs (for instance

institutional

and

test

in a specific order one should overwrite the other when it comes to certain values? For instance, shouldnt the

test

profile's

resourcelimits

overwrite the ones specified in the institutional config if the

test

profile is specified later? Or am I misunderstanding

Hey! I'm trying to dynamically set a scale_factor for deeptools_bamcoverage. Failing so far. Has anyone achieved this?

Is it still recommended/necessary to run

.first()

after mixing module versions into

ch_versions

? Given

.unique

is run prior passing to MultiQC, is there any overhead benefit for taking just the

version.yaml

from the first module invocation vs passing all

versions.yml

and running unique

Workflow gets hung up on subworkflows .view() call. View() works fine outside of it, How i am calling the subworkflow

def longpac_longpolish = SAMPLESHEETFILTERING.out.list_longpac_longPolish
    def flattened_result = longpac_longpolish
        .filter { value -> value instanceof List && !value.isEmpty() }
        .flatMap()
    flattened_result.view()
PACBIO_SUBWORKFLOW(flattened_result)

it views() fine, emitting [[id:Sample1, polish:long, basecaller:NA], short1NA, short2NA, TestDatasetNfcore/Pacbio_illuminaPolish/PacbioSRR27591472.hifi.fastq.gz, assemblyNA] [[id:Sample2, polish:long, basecaller:NA], short1NA, short2NA, TestDatasetNfcore/Pacbio_illuminaPolish/PacbioSRR27591472.hifi.fastq.gz, assemblyNA] but when i pass it to the subworkflow

workflow PACBIO_SUBWORKFLOW {

    take:
    ch_input_full // channel: [ val(meta), files/data, files/data, files/data..etc ]
    // bam_file
    // polish
    // gambitdb
    // krakendb

    main:
    def ch_output = Channel.empty()
    def ch_versions = Channel.empty()
    println("hello")
    ch_input_full.view()

It just prints hello, and gets hung up, doesn't seem to ever print the channel values? just sits there. I dont understand why? my nextflow.log also says all processes finished, all barriers passed

Aug-22 13:23:17.907 [main] DEBUG nextflow.script.ScriptRunner - > Awaiting termination 
Aug-22 13:23:17.907 [main] DEBUG nextflow.Session - Session await
Aug-22 13:23:17.907 [main] DEBUG nextflow.Session - Session await > all processes finished
Aug-22 13:23:17.908 [main] DEBUG nextflow.Session - Session await > all barriers passed

As we know process names get truncated with ansi output. I know I can get full names if I use

-ansi-log false

but I do want the ansi output to have the latest colored output. I saw in this issue @Phil Ewels was suggesting something with full names as in the pic. Was this implemented? Can I get something like that with ansi logs and full process name or at least longer? As it is, it's still very challenging to read

if i have 2 channels from channel factories ch_1 = Channel.of(1,2,3) ch_2 = Channel.of(4,5,6) if i input them into a subworkflow together subworkflow(ch_1, ch_2) will they always emit in order? so the first value of ch_1 will be with the first value of ch_2 and so on?