Skip to content

perf: speed up binning by not waiting for all mapping jobs - #1061

Merged
dialvarezs merged 2 commits into
nf-core:devfrom
dialvarezs:feat/per-assembly-mapping
Jun 27, 2026
Merged

perf: speed up binning by not waiting for all mapping jobs#1061
dialvarezs merged 2 commits into
nf-core:devfrom
dialvarezs:feat/per-assembly-mapping

Conversation

@dialvarezs

@dialvarezs dialvarezs commented Jun 27, 2026

Copy link
Copy Markdown
Member

This is performance/QoL improvement.

Right now, binners have to wait for every BOWTIE2_ASSEMBLY_ALIGN / MINIMAP2_ASSEMBLY_ALIGN to finish, because groupTuple(by:0) doesn't know how many elements will be in the tuple.

In practice this means a couple of straggler alignment jobs stall the entire shortread (or longread) binning path, even for assemblies whose mappings already completed. This inflates runtimes and makes resource usage suboptimal.

Fix: the mappings per assembly are predictable from binning_map_mode (own -> 1, group -> samples in group, all -> total samples). We compute that count and wrap the key with groupKey(key, count), so each group is released as soon as it's complete.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@dialvarezs dialvarezs changed the title feat: use groupKey so mappings group per-assembly feat:sSpeed up binning by not waiting for all mapping jobs Jun 27, 2026
@dialvarezs dialvarezs changed the title feat:sSpeed up binning by not waiting for all mapping jobs feat: speed up binning by not waiting for all mapping jobs Jun 27, 2026
@dialvarezs dialvarezs changed the title feat: speed up binning by not waiting for all mapping jobs perf: speed up binning by not waiting for all mapping jobs Jun 27, 2026
@dialvarezs
dialvarezs merged commit e04ab12 into nf-core:dev Jun 27, 2026
24 checks passed
@dialvarezs
dialvarezs deleted the feat/per-assembly-mapping branch June 27, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants