Description of the bug
On -resume, the PRELUDE:PRELUDE_SUMMARY_* steps re-run every time, even when nothing upstream has changed.
It traces to subworkflows/local/prelude/main.nf, where the summary inputs are JSON files built on the fly inside the .map{} operators, e.g.
def f = file("$meta".md5().toString() + ".json")
f.write(new groovy.json.JsonBuilder(meta).toString())
The name and contents are stable, but the file gets rewritten on every run, so its last-modified time changes — and Nextflow's default input hashing keys on name+size+mtime, so it's a cache miss each time.
On local/SLURM this is harmless (the tasks take ~1s). On AWS Batch it isn't: each re-run has to be scheduled onto an instance (queue → scale-up → image pull), so a 1-second task shows a ~5-minute duration. Every resume then pays several minutes of pure scheduling overhead for work that didn't change, which eats into the whole point of resuming past the expensive ASHLAR/segmentation steps.
Fix would be to set cache 'lenient' on the PRELUDE summary processes (lenient hashing ignores mtime, so a stable name+size caches correctly), or to stop generating the input files inside the operators.
Command used and terminal output
Relevant files
No response
System information
No response
Description of the bug
On -resume, the
PRELUDE:PRELUDE_SUMMARY_*steps re-run every time, even when nothing upstream has changed.It traces to subworkflows/local/prelude/main.nf, where the summary inputs are JSON files built on the fly inside the
.map{}operators, e.g.The name and contents are stable, but the file gets rewritten on every run, so its last-modified time changes — and Nextflow's default input hashing keys on name+size+mtime, so it's a cache miss each time.
On local/SLURM this is harmless (the tasks take ~1s). On AWS Batch it isn't: each re-run has to be scheduled onto an instance (queue → scale-up → image pull), so a 1-second task shows a ~5-minute duration. Every resume then pays several minutes of pure scheduling overhead for work that didn't change, which eats into the whole point of resuming past the expensive ASHLAR/segmentation steps.
Fix would be to set
cache 'lenient'on the PRELUDE summary processes (lenient hashing ignores mtime, so a stable name+size caches correctly), or to stop generating the input files inside the operators.Command used and terminal output
Relevant files
No response
System information
No response