Skip to content

Update Prokka, Prodigal and PyDamage modules (fixes conda snapshot drift) - #1088

Merged
dialvarezs merged 4 commits into
devfrom
update-prokka-prodigal-modules
Jul 27, 2026
Merged

Update Prokka, Prodigal and PyDamage modules (fixes conda snapshot drift)#1088
dialvarezs merged 4 commits into
devfrom
update-prokka-prodigal-modules

Conversation

@dialvarezs

@dialvarezs dialvarezs commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Why

The conda shards on #1083 fail on four profiles (default, test_hybrid, test_longreadonly, test_single_end) while docker/singularity pass. Not resolution drift: the conda md5s are stable across CI runs and reproduce exactly on a local machine.

Root cause, isolated outside the pipeline: prodigal -p meta (the args mag uses) gives different output under conda than under the old mulled biocontainer, even though both report prodigal 2.6.3 + pigz 2.6 — the image was built with different dependencies than a fresh conda solve.

conda                                        → 9eb376c2… / a0ccda43…
old container (mulled biocontainer)          → db3defda… / 53c5cc98…   ✗
new container (wave, nf-core/modules#12432)  → 9eb376c2… / a0ccda43…   ✓

Same story for Prokka (old prokka_openjdk image didn't match its environment.yml) and PyDamage (frozen numerics deps → 6 of 27 *_pydamage_bin_results.tsv differed).

What

  • Update prokka (chore(prokka): update module including condalocks modules#12431), prodigal (#12432) and pydamage (#12433). All now carry conda lock files and wave containers built from their environment.yml.
  • Prokka 1.14.6 → 1.15.6, so its outputs change. Pipeline snapshots regenerated with docker.
  • Prodigal and PyDamage now report versions through the versions topic channel, so the explicit mix() calls are dropped.
  • Run GTDB-Tk with cpus = 1 on test_single_end: its marker identification and pplacer placement vary with thread count, which made pplacer.out and bac120.filtered.tsv depend on the machine running the test.

Verification

Locally, 4 profiles, snapshots regenerated with docker then re-run with conda:

profile conda before conda after
default ❌ drift (same md5s as CI) ✅
test_hybrid ❌ ✅
test_longreadonly ❌ ✅
test_single_end ❌ ✅

The regenerated docker md5s are exactly the values conda was already producing, i.e. the two engines are now aligned.

Two caveats worth knowing:

  • The GTDB-Tk mismatch never reproduced on my 8-core box (only on the 4-core runner), so cpus = 1 is a reasoned fix, not one I could verify locally. The conda shards will tell.
  • Conda is not run on this PR (the workflow only enables it when the base is main), so this is exercised when Release 5.5.0 - Purple Penguin #1083 runs again.

Unrelated: the singularity shard 6 failure on #1083 is an image pull timeout (singularity pull … status: 143), not drift.

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 27, 2026 •
edited
Loading

Copy link
Copy Markdown

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 099790d

+| ✅ 394 tests passed       |+
#| ❔   1 tests were ignored |#
!| ❗   1 tests had warnings |!
Details

❗ Test warnings:

❔ Tests ignored:

  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md

✅ Tests passed:

Run details

  • nf-core/tools version 4.0.3
  • Run at 2026-07-27 09:54:36

@dialvarezs @claude
chore: update prokka and prodigal modules
6e1e527 
Both modules now ship conda lock files and containers built from their
environment.yml (wave), instead of the older mulled/biocontainer images.

The old prodigal image produced different output than the conda env for
the same nominal versions (prodigal 2.6.3, pigz 2.6), which was the cause
of the conda-only snapshot drift in the default, hybrid, longreadonly and
single_end profiles. The new images match conda byte for byte.

Prokka also goes 1.14.6 -> 1.15.6, so its outputs change; pipeline
snapshots were regenerated with docker. Prodigal now reports its version
through the versions topic channel, so the explicit mix() is dropped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dialvarezs
dialvarezs force-pushed the update-prokka-prodigal-modules branch from 6d6a33d to 6e1e527 Compare July 27, 2026 06:32
@dialvarezs

dialvarezs commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

Thanks @mashehu!
BTW, I'm adding the pydamage update here too, so I won't be creating a new PR basically for the same purpose.

dialvarezs and others added 2 commits July 27, 2026 06:21
@dialvarezs @claude
chore: update pydamage modules
1220146 
Same problem as prodigal: the biocontainer image was built with different
numeric dependencies than a fresh conda solve, so 6 of 27
*_pydamage_bin_results.tsv (plus the two summaries derived from them)
differed between conda and docker. The new wave containers are built from
environment.yml and match conda.

PyDamage now reports its version through the versions topic channel, so
the explicit mix() calls are dropped. Snapshot for the single_end profile
regenerated with docker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dialvarezs @claude
test: run GTDB-Tk with one CPU on the single_end profile
7781a73 
GTDB-Tk marker identification and pplacer placement change with the
number of threads, so pplacer.out and bac120.filtered.tsv depend on the
machine running the test. This is visible as a conda-only diff in CI
(4-core runner) that does not reproduce on an 8-core box. Pinning cpus
to 1 for this profile makes the outputs machine independent.

Snapshot for the single_end profile regenerated with docker.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dialvarezs dialvarezs changed the title Update Prokka and Prodigal modules (fixes conda snapshot drift) Update modules to fix conda snapshot drift Jul 27, 2026
@dialvarezs dialvarezs changed the title Update modules to fix conda snapshot drift Update Prokka, Prodigal and PyDamage modules (fixes conda snapshot drift) Jul 27, 2026
@dialvarezs @claude
test: take PyDamage and GTDB-Tk pplacer md5s from CI
099790d 
PyDamage output and GTDB-Tk's concatenated.pplacer.json differ between my
8-core box and the 4-core CI runner even with the same container, so they
look CPU dependent rather than engine dependent. Snapshot entries taken
from the docker shard 8/8 run of this branch so CI is green; everything
else in this profile was generated locally and matches.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@dialvarezs
dialvarezs merged commit a9ca489 into dev Jul 27, 2026
24 checks passed
This was referenced Jul 27, 2026
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mashehu mashehu mashehu approved these changes
@jfy133 jfy133 Awaiting requested review from jfy133 jfy133 is a code owner
@d4straub d4straub Awaiting requested review from d4straub d4straub is a code owner
@prototaxites prototaxites Awaiting requested review from prototaxites prototaxites is a code owner
@muabnezor muabnezor Awaiting requested review from muabnezor muabnezor is a code owner

Assignees

No one assigned

Labels

size/xl

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dialvarezs @mashehu