Skip to content

feat(template): add process_low_memory resource label - #4264

Merged
pinin4fjords merged 3 commits into
nf-core:devfrom
pinin4fjords:pinin4fjords/process-low-memory-label
May 6, 2026
Merged

feat(template): add process_low_memory resource label#4264
pinin4fjords merged 3 commits into
nf-core:devfrom
pinin4fjords:pinin4fjords/process-low-memory-label

Conversation

@pinin4fjords

Copy link
Copy Markdown
Member

Summary

Adds a process_low_memory resource label to the pipeline-template base.config that overrides only memory, mirroring the existing process_high_memory modifier.

withLabel:process_low_memory {
    memory = { 1.GB * task.attempt }
}

Motivation

There's currently no way to express "cpu-bound but memory-light" via stacked labels. process_high pins both cpus and memory at 12 / 72 GB; dropping to process_low gives up the cpus too. Modern Rust streaming tools (e.g. trim_galore 2.x, with ~100 MB peak_rss on 30M PE) want exactly that shape.

With process_low_memory added, a module can stack labels (precedence does what you'd expect - later labels override earlier overrides):

process FOO {
    label 'process_high'
    label 'process_low_memory'
    ...
}

→ 12 cpus * task.attempt, 1 GB * task.attempt, 16 h * task.attempt.

This is the symmetric counterpart of the existing pattern that process_high_memory already enables (low cpus + high memory for memory-hungry single-threaded tools).

Discussed in #nf-core Slack on 2026-05-05 - Phil Ewels suggested the label name; James Fellows Yates +1'd more granularity for resource optimisation.

Changes

  • nf_core/pipeline-template/conf/base.config - new withLabel:process_low_memory block (1 GB * task.attempt).
  • nf_core/modules/lint/main_nf.py - add to correct_process_labels so module lint accepts it.
  • nf_core/components/create.py - add to the autocomplete list shown by nf-core modules create.
  • CHANGELOG.md - entries under Linting and Template.

Why 1 GB?

process_single is 6 GB, process_low is 12 GB. The new label needs to be a meaningful step down from process_single to be worth having - 1 GB doubles to 2 GB on retry, then 3 GB, before errorStrategy kicks in, which is comfortable for tools that genuinely use < 500 MB. Open to bumping to 2 GB or 3 GB if reviewers prefer a more conservative floor.

Test plan

  • Existing module lint tests still pass.
  • nf-core modules create shows the new label in the autocomplete list.
  • A pipeline using label 'process_high' + label 'process_low_memory' resolves to 12 cpus + 1 GB on the first attempt.

Out of scope

process_high_cpu, process_low_cpu - James floated more axes. Worth a follow-up but keeping this PR small.

Adds a memory-only override label that mirrors the existing
process_high_memory pattern, so it can be stacked with one of the
cpu/time bands (e.g. `process_high` + `process_low_memory` to
get lots of cpus + tight memory ceiling).

Motivation: tools that are cpu-bound but memory-light (Rust streaming
binaries, e.g. trim_galore 2.x ~100 MB peak_rss) had no way to express
that resource shape with the existing labels - process_high pinned both
cpus and memory at 12/72GB, while dropping to process_low gave up the
cpus too.

Wires the new label into:
- pipeline-template base.config (`memory = { 1.GB * task.attempt }`)
- modules lint allowlist
- nf-core modules create label autocomplete

Discussed in #nf-core Slack #help thread on 2026-05-05.
@codecov

codecov Bot commented May 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 77.81%. Comparing base (b291b0b) to head (1903ddb).
⚠️ Report is 4 commits behind head on dev.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ewels ewels left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Should be fairly non-destructive. Can't imagine many clusters will want / need to customise or optimise this label, so it really is just to knock down otherwise excessive memory requirements.

Will need a corresponding documentation update on the website.

@pinin4fjords

Copy link
Copy Markdown
Member Author

Merging per Slack discussion: https://nfcore.slack.com/archives/C043UU89KKQ/p1778073031820339?thread_ts=1777997824.867769&cid=C043UU89KKQ

@pinin4fjords
pinin4fjords enabled auto-merge May 6, 2026 13:18
@pinin4fjords
pinin4fjords merged commit b0eef1d into nf-core:dev May 6, 2026
113 of 118 checks passed
SPPearce pushed a commit to atrigila/modules that referenced this pull request May 7, 2026
…ore#11541)

* fix(trimgalore): stack process_low_memory on top of process_low

Now that nf-core/tools#4264 has merged and the pipeline-template
`base.config` defines `process_low_memory` (1 GB * task.attempt),
the trimgalore module can drop its memory ceiling without giving up
the cpu/time bands set by process_low.

Composed budget on first attempt:
- cpus    = 2     (process_low; 1 worker thread paired)
- memory  = 1 GB  (process_low_memory; ~10x observed peak_rss of ~100 MB on 30M PE)
- time    = 4 h   (process_low; ~160x observed runtime of ~1.5 min on 30M PE)

The tool itself caps trim_galore --cores at 8 worker threads, so
allocating more cpus is wasted unless paired with very large inputs;
pipelines that genuinely need that throughput can override per-process
or stack `process_high` instead. This is the conservative default
that satisfies Felix's original 100 MB / 10 min ask while keeping
existing 1-worker-paired throughput intact for typical use.

* fix(trimgalore): drop redundant process_low label

The template's process default is cpus=1 / time=4.h, which is what
process_low_memory already inherits when no other resource label is
applied. process_low's cpus=2 changes nothing useful here:
trim_galore's --cores formula (`cores = max(1, task.cpus - 4)` paired)
saturates at 1 worker thread for any cpus <= 5, so 1 vs 2 cpus gives
the same worker count. Drop the redundant label.

* fix(trimgalore): use process_medium for cpus, keep process_low_memory

Stack process_medium with process_low_memory so the cpu allocation
is enough to give trim_galore 2 paired worker threads on production
data, while keeping the memory ceiling tight at 1 GB.

trim_galore's --cores formula derives worker count from task.cpus:
  cores = max(1, task.cpus - 4)  // paired
  cores = max(1, task.cpus - 3)  // single (capped at 8)

Implications:
- cpus = 1 (default)        -> 1 worker thread paired
- cpus = 2 (process_low)    -> 1 worker thread paired (no gain)
- cpus = 6 (process_medium) -> 2 worker threads paired (~2x throughput)
- cpus = 12 (process_high)  -> 8 worker threads paired (saturates the cap)

process_medium is the cheapest band that crosses the 1->2 worker
threshold, where actual trimming throughput first improves. Useful for
real production inputs (100M+ PE) where the extra worker materially
shortens wall time; small test data won't notice the difference. Memory
stays at process_low_memory's 1 GB regardless.
JoseEspinosa pushed a commit to JoseEspinosa/website that referenced this pull request May 7, 2026
Companion to nf-core/tools#4264, which adds a process_low_memory
resource label to the pipeline-template base.config. Updates the
modules resource-requirements spec to:

- list all bundled and modifier labels currently shipped in the
  template (process_single, process_low, process_medium, process_high
  and the modifier-only process_long, process_low_memory,
  process_high_memory)
- document the label-stacking pattern that the modifier labels enable,
  with an example showing process_high + process_low_memory for
  CPU-bound but memory-light tools

Also unpins the base.config link from a line range that already drifted
out of date and switches the branch reference from master to main.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants