Skip to content

Pipeline fails due to file errors - missing files #582

@barkdaron

Description

@barkdaron

I have succeeded in installing nextflow and quantms - not that hard and I feel confident that the problem is not there. I have been testing the pipeline with the suggested test
nextflow run bigbio/quantms -profile test_lfq,apptainer

The pipe line will do something and eventually fail. This is an example of the error:
Error - cannot open parameter file "/tmp/20250910_181120_compute01_41_1/param.txt".
Process '/opt/OpenMS/thirdparty/Comet/comet.exe' did not finish successfully (exit code: 1). Please check the log.
There is nothing interesting in the log..
I observed that by repeating the pipeline with -resume I would progress a bit every time and eventually finish.
I have been thinking the problem is grounded in IO race conditions. I did fix this particular error (or I have not seen it repeated) by adding a 'sync' to the modules/local/openms/comet/main.nf

A new error has since popped up:
Could not remove directory 20250911_104006_compute01_42_1!
Error: File not found (the file '/tmp/20250911_104006_compute01_42_1/result.pep.xml' could not be found) The observation is the same: Repeating with -resume the pipeline will eventually finish successfully and the problem/situation can be explained by IO race condition.

I have also remounted my disk to sync option instead of async, which I believed would fix a race condition, but the difference in pipeline behavior was difficult to spot.

I am using a modern Dell server with ubuntu server 24.04.3 LTS. It is part of a cluster.
I realize that nextflow and quantms is mature software, and should not suffer from these kind of problems, so I am reaching out in the hope that this is a known issue, which I simply could not google.
BTW: I have a chatgpt conversation about the issue, which I won't publish in this too public forum, but that I don't mind sharing with individuals.

Regards
Peter

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions