feat(template): add process_low_memory resource label - #4264
Merged
pinin4fjords merged 3 commits intoMay 6, 2026
Merged
Conversation
Adds a memory-only override label that mirrors the existing
process_high_memory pattern, so it can be stacked with one of the
cpu/time bands (e.g. `process_high` + `process_low_memory` to
get lots of cpus + tight memory ceiling).
Motivation: tools that are cpu-bound but memory-light (Rust streaming
binaries, e.g. trim_galore 2.x ~100 MB peak_rss) had no way to express
that resource shape with the existing labels - process_high pinned both
cpus and memory at 12/72GB, while dropping to process_low gave up the
cpus too.
Wires the new label into:
- pipeline-template base.config (`memory = { 1.GB * task.attempt }`)
- modules lint allowlist
- nf-core modules create label autocomplete
Discussed in #nf-core Slack #help thread on 2026-05-05.
Codecov Report✅ All modified and coverable lines are covered by tests. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
ewels
approved these changes
May 5, 2026
ewels
left a comment
Member
There was a problem hiding this comment.
LGTM!
Should be fairly non-destructive. Can't imagine many clusters will want / need to customise or optimise this label, so it really is just to knock down otherwise excessive memory requirements.
Will need a corresponding documentation update on the website.
Member
Author
|
Merging per Slack discussion: https://nfcore.slack.com/archives/C043UU89KKQ/p1778073031820339?thread_ts=1777997824.867769&cid=C043UU89KKQ |
pinin4fjords
enabled auto-merge
May 6, 2026 13:18
This was referenced May 6, 2026
SPPearce
pushed a commit
to atrigila/modules
that referenced
this pull request
May 7, 2026
…ore#11541) * fix(trimgalore): stack process_low_memory on top of process_low Now that nf-core/tools#4264 has merged and the pipeline-template `base.config` defines `process_low_memory` (1 GB * task.attempt), the trimgalore module can drop its memory ceiling without giving up the cpu/time bands set by process_low. Composed budget on first attempt: - cpus = 2 (process_low; 1 worker thread paired) - memory = 1 GB (process_low_memory; ~10x observed peak_rss of ~100 MB on 30M PE) - time = 4 h (process_low; ~160x observed runtime of ~1.5 min on 30M PE) The tool itself caps trim_galore --cores at 8 worker threads, so allocating more cpus is wasted unless paired with very large inputs; pipelines that genuinely need that throughput can override per-process or stack `process_high` instead. This is the conservative default that satisfies Felix's original 100 MB / 10 min ask while keeping existing 1-worker-paired throughput intact for typical use. * fix(trimgalore): drop redundant process_low label The template's process default is cpus=1 / time=4.h, which is what process_low_memory already inherits when no other resource label is applied. process_low's cpus=2 changes nothing useful here: trim_galore's --cores formula (`cores = max(1, task.cpus - 4)` paired) saturates at 1 worker thread for any cpus <= 5, so 1 vs 2 cpus gives the same worker count. Drop the redundant label. * fix(trimgalore): use process_medium for cpus, keep process_low_memory Stack process_medium with process_low_memory so the cpu allocation is enough to give trim_galore 2 paired worker threads on production data, while keeping the memory ceiling tight at 1 GB. trim_galore's --cores formula derives worker count from task.cpus: cores = max(1, task.cpus - 4) // paired cores = max(1, task.cpus - 3) // single (capped at 8) Implications: - cpus = 1 (default) -> 1 worker thread paired - cpus = 2 (process_low) -> 1 worker thread paired (no gain) - cpus = 6 (process_medium) -> 2 worker threads paired (~2x throughput) - cpus = 12 (process_high) -> 8 worker threads paired (saturates the cap) process_medium is the cheapest band that crosses the 1->2 worker threshold, where actual trimming throughput first improves. Useful for real production inputs (100M+ PE) where the extra worker materially shortens wall time; small test data won't notice the difference. Memory stays at process_low_memory's 1 GB regardless.
JoseEspinosa
pushed a commit
to JoseEspinosa/website
that referenced
this pull request
May 7, 2026
Companion to nf-core/tools#4264, which adds a process_low_memory resource label to the pipeline-template base.config. Updates the modules resource-requirements spec to: - list all bundled and modifier labels currently shipped in the template (process_single, process_low, process_medium, process_high and the modifier-only process_long, process_low_memory, process_high_memory) - document the label-stacking pattern that the modifier labels enable, with an example showing process_high + process_low_memory for CPU-bound but memory-light tools Also unpins the base.config link from a line range that already drifted out of date and switches the branch reference from master to main.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a
process_low_memoryresource label to the pipeline-templatebase.configthat overrides onlymemory, mirroring the existingprocess_high_memorymodifier.Motivation
There's currently no way to express "cpu-bound but memory-light" via stacked labels.
process_highpins both cpus and memory at 12 / 72 GB; dropping toprocess_lowgives up the cpus too. Modern Rust streaming tools (e.g. trim_galore 2.x, with ~100 MB peak_rss on 30M PE) want exactly that shape.With
process_low_memoryadded, a module can stack labels (precedence does what you'd expect - later labels override earlier overrides):→ 12 cpus * task.attempt, 1 GB * task.attempt, 16 h * task.attempt.
This is the symmetric counterpart of the existing pattern that
process_high_memoryalready enables (low cpus + high memory for memory-hungry single-threaded tools).Discussed in #nf-core Slack on 2026-05-05 - Phil Ewels suggested the label name; James Fellows Yates +1'd more granularity for resource optimisation.
Changes
nf_core/pipeline-template/conf/base.config- newwithLabel:process_low_memoryblock (1 GB * task.attempt).nf_core/modules/lint/main_nf.py- add tocorrect_process_labelsso module lint accepts it.nf_core/components/create.py- add to the autocomplete list shown bynf-core modules create.CHANGELOG.md- entries under Linting and Template.Why 1 GB?
process_singleis 6 GB,process_lowis 12 GB. The new label needs to be a meaningful step down fromprocess_singleto be worth having - 1 GB doubles to 2 GB on retry, then 3 GB, beforeerrorStrategykicks in, which is comfortable for tools that genuinely use < 500 MB. Open to bumping to 2 GB or 3 GB if reviewers prefer a more conservative floor.Test plan
nf-core modules createshows the new label in the autocomplete list.label 'process_high'+label 'process_low_memory'resolves to 12 cpus + 1 GB on the first attempt.Out of scope
process_high_cpu,process_low_cpu- James floated more axes. Worth a follow-up but keeping this PR small.