Add PyPOLCA polishing and use polished assemblies for long-read binning by Harshita-sriv · Pull Request #1048 · nf-core/mag

Harshita-sriv · 2026-06-02T07:23:33Z

Summary

This PR adds a PyPOLCA polishing module and integrates polished assemblies into the long-read MAG workflow.

Changes

Added a new PyPOLCA module and conda environment.
Added publication of PyPOLCA outputs.
Added the run_pypolca parameter.
Modified the workflow so that long-read binning uses ch_polished_assemblies instead of raw ch_longread_assemblies.

dialvarezs · 2026-06-02T07:36:22Z

Hi @Harshita-sriv, thanks for this!
For the meantime, can you join the nf-core org? That way you will able to run the ci without explicit approval.
https://nf-co.re/join

dialvarezs

In general I think it looks right, but needs to migrate to the nf-core module, and of course, pass the ci.

Harshita-sriv · 2026-06-02T08:55:53Z

Thanks for the review. I investigated the CI failures and found that the current implementation does not define ch_polished_assemblies when run_pypolca is not enabled, which causes the test workflows to fail. I'll address this together with the requested migration to the official nf-core PyPOLCA module, removal of debug statements, and formatting fixes

Merge remote-tracking branch 'upstream/dev' into add-polishing-module

dialvarezs · 2026-06-04T11:33:30Z

Hi @Harshita-sriv! I think this looks fine, but it would be ideal to wire it up in some test. I'll think what would be the best option here and I'll come back to you.
For the meantime, can you update the changelog?
Thanks!

d4straub

Thanks for the addition, I think thats absolutely desired!
Looks good but I have some concerns with multiple samples.

Harshita-sriv · 2026-06-10T09:13:21Z

I noticed that the remaining lint failure is due to modifications in modules/nf-core/pypolca/run, which no longer matches the upstream module.

The changes were introduced because PyPOLCA was failing during testing, and with the modifications the pipeline completes successfully on the test dataset.

Would it be acceptable to keep these modifications to the PyPOLCA module, or would you prefer that the implementation uses the upstream module unchanged and handles the required logic elsewhere?

d4straub · 2026-06-11T07:40:07Z

Would it be acceptable to keep these modifications to the PyPOLCA module, or would you prefer that the implementation uses the upstream module unchanged and handles the required logic elsewhere?

Some of the changes are not acceptable: the output block and the versioning must be kept as it was originally. Additionally the line ch_versions = ch_versions.mix(PYPOLCA_RUN.out.versions) in workflows/mag.nf has to be removed.
The incoming data (contigs) should be compressed as in the original code. Even if it would be necessary to allow non-compressed input (which may be ok in same cases), copying the files when already uncompressed seems like an unnecessary burden on IO and file sizes to me (assemblies in nf-core/mag can be huge!).

Generally, small changes on nf-core modules can be done and accepted by the linting via https://nf-co.re/docs/nf-core-tools/cli/modules/patch

Harshita-sriv · 2026-06-12T06:27:20Z

@d4straub I restored the PyPOLCA module to the upstream nf-core implementation and moved the coassembly-specific logic into the workflow layer (workflows/mag.nf). The workflow now groups short reads appropriately when --coassemble_group is enabled before passing them to PyPOLCA.

The remaining CI issue appears to be the check_local_copy lint check for pypolca/run. Locally, the module matches the installed nf-core module definition (nf-core modules info pypolca/run), and the workflow compiles and reaches PYPOLCA_RUN in stub runs. All other CI checks pass.

Could you advise whether there is an additional module sync/patch step expected for this module?

d4straub · 2026-06-12T06:45:52Z

Looks good.
I think the easiest way of satisfying the linting is to re-install the module.

dialvarezs

Hi again @Harshita-sriv! This is looking good!

A few last things:

Regarding the testing setup:
I think the best option is to use the longreadonly_alternatives but changing the samplesheet so it uses the hybrid one that has LR and SR data. Then disable SPAdes, SPAdes hybrid and MEGAHIT at least. That way we can get this tested in that profile.

Also I left a comment regarding the pypolca module requiring gzipped input (that will need updating the nf-core module).

Harshita-sriv · 2026-06-16T03:35:39Z

Hey @dialvarezs

I'm still seeing the remaining CI failures, and I haven't been able to reproduce/update the snapshots locally. Since you made the latest changes, could you take a look and let me know whether there's anything else I should update on my side?

dialvarezs · 2026-06-16T13:50:31Z

Hi @Harshita-sriv, it just needed to regenerate the snapshot and exclude the VCF, because it has the date.
I also realized that we missed one point: Nextflow channels are not guaranteed to be emited in the same order. CI pass because it's only one sample. I fixed that with a combine and multiMap mix.

dialvarezs · 2026-06-16T13:51:25Z

For me this looks good to merge now. Can you give it a final look @d4straub?

Harshita-sriv · 2026-06-16T16:34:25Z

Hey, @dialvarezs!
Thanks for fixing those issues and for the explanation! I'll take a look through the changes to understand them better. Thanks again for the review.

d4straub

I think I still found some relevant inconsistencies.

Harshita-sriv · 2026-06-18T06:27:32Z

Hey @d4straub and @dialvarezs,

I've addressed the review comments, updated MAG to the latest merged PyPOLCA module revision, and regenerated the affected snapshots. The longreadonly_alternatives test passes locally with Docker.

I had to rerun the CI jobs after canceling a workflow while investigating a long-running NF-test shard. The CI checks have now been completed successfully. 🎉

Do you think this is good to go? Please say yes! 😄

d4straub · 2026-06-18T06:42:25Z

+                "Assembly/PYPOLCA/minigut/minigut/minigut.report",
+                "Assembly/PYPOLCA/minigut/minigut/minigut.vcf",
+                "Assembly/PYPOLCA/minigut/minigut/minigut_pypolca.fasta"


I am still a bit concerned here: In this current test, only one assembler (FLYE) is run, produces "Assembly/FLYE/FLYE-minigut.assembly.fasta.gz" and corrected to file "Assembly/PYPOLCA/minigut/minigut/minigut_pypolca.fasta". But if two assemblers are run, FLYE and metaMDBG, both corrected assemblies will be named to "Assembly/PYPOLCA/minigut/minigut/minigut_pypolca.fasta", overwriting each other.
I think the file name must include the assembler. Or did I get something wrong?

Done with the changes. meta2.assembler came in handy! I've updated the naming to preserve the assembler in the PyPOLCA outputs and regenerated the snapshots.

dialvarezs · 2026-06-18T08:53:45Z

+        ch_polished_assemblies = PYPOLCA_RUN.out.polished
+            .map { meta, polished ->
+                [meta.id, polished]
+            }
+            .join(
+                ch_longread_assemblies.map { meta, assembly ->
+                    [meta.id, meta]
+                }
+            )
+            .map { id, polished, meta ->
+                [meta, polished]
+            }


If there are multiple assemblers enabled, this join could be problematic. We're not using the assembler for the join, so, if we have two assemblies for the same sample, the join will do a cross product, producing 4 outputs instead of 2 (each combination of original assembler + poslished one).

The problem is that pypolca emits the reads meta, not the assembly one, so we can't get that info easily.

In my opinion, we should update the module so it emits the meta from the assembly, not the one from the reads. The output is more closely related to the assembly than the reads, so it makes more sense. If we have that change, we can remove this whole block, because we will have the meta from the assembly here already, and will prevent the issue I'm describing.

Well, I guess in this case I would just swap the inputs: make the assembly the first, because it's the primary one and leave the reads as the second input.
The other option is using meta2 in the outputs, but I think this feels like a poorer design.

I've refactored the module to use the assembly metadata as the primary input, updated the MAG integration/tests, and verified the output structure locally. CI is now passing.

d4straub

No further comments, looks perfect now!
That was quite a journey, but a successful one ;)

dialvarezs

Hi @Harshita-sriv!

As a finishing touches I updated some variable names and moved some things around to improve semantics, and also updated output docs.
I think this is good to merge now.

You did a wonderful job! Thanks for contributing this!

Feel free to push the merge button :D

Add PyPOLCA polishing and use polished assemblies for long-read binning

2f7b365

Harshita-sriv requested review from d4straub, dialvarezs, jfy133, muabnezor and prototaxites as code owners June 2, 2026 07:23

dialvarezs requested changes Jun 2, 2026

View reviewed changes

Comment thread modules/local/pypolca/main.nf Outdated

Comment thread workflows/mag.nf Outdated

Comment thread workflows/mag.nf

harshita.s and others added 6 commits June 2, 2026 17:44

Use official nf-core PyPOLCA module

d5daf4b

t push origin add-polishing-module

05fc21f

Merge remote-tracking branch 'upstream/dev' into add-polishing-module

Fix unresolved merge conflict in modules.json

1c14fa0

Remove obsolete local PyPOLCA module and update snapshots

2e28a41

Fix snapshot ordering for Flye assembly_info output

a40cce8

Apply pre-commit formatting

93e58c0

Harshita-sriv requested a review from dialvarezs June 3, 2026 09:58

d4straub reviewed Jun 5, 2026

View reviewed changes

Comment thread nextflow_schema.json Outdated

Comment thread workflows/mag.nf Outdated

Comment thread conf/modules.config Outdated

harshita.s and others added 3 commits June 10, 2026 12:39

Update PyPOLCA integration

e022822

Aply pre-commit formatting

c3a6998

Remove unused paramsHelp import

6579f8d

Fix PyPOLCA integration for coassembly support

ec34498

Sync pypolca module with nf-core/modules

df42101

dialvarezs reviewed Jun 15, 2026

View reviewed changes

Comment thread modules/nf-core/pypolca/run/main.nf Outdated

Harshita-sriv added 2 commits June 15, 2026 10:27

Add PyPolca support to longread alternatives test

d62c043

chore: update pypolca module

6a5b25c

chore: formatting

583c07f

dialvarezs added 3 commits June 16, 2026 08:48

chore: update snapshots

2ad7c2f

chore: ignore pypolca vcf

29bd56e

fix: add combine step to pair pypolca inputs correctly

c6f518e

d4straub reviewed Jun 17, 2026

View reviewed changes

Comment thread tests/test_longreadonly_alternatives.nf.test.snap Outdated

Comment thread tests/test_longreadonly_alternatives.nf.test.snap Outdated

Comment thread workflows/mag.nf

fix: preserve assembly metadata after pypolca polishing

9606089

dialvarezs reviewed Jun 17, 2026

View reviewed changes

Comment thread conf/modules.config Outdated

Harshita-sriv force-pushed the add-polishing-module branch from 2056f30 to 5b6382a Compare June 17, 2026 15:23

chore: rename pypolca polished assembly output

c92845b

Harshita-sriv force-pushed the add-polishing-module branch from 5b6382a to c92845b Compare June 17, 2026 15:32

test: update snapshots for pypolca outputs

69c5e4b

dialvarezs reviewed Jun 17, 2026

View reviewed changes

Comment thread tests/test_longreadonly_alternatives.nf.test.snap Outdated

fix: update pypolca module and snapshots

d42bdbb

d4straub reviewed Jun 18, 2026

View reviewed changes

Comment thread conf/modules.config

fix: preserve assembler name in pypolca outputs

58ed6ca

dialvarezs reviewed Jun 18, 2026

View reviewed changes

refactor: update pypolca module interface

03b1a05

Harshita-sriv force-pushed the add-polishing-module branch from 8fab6cd to 03b1a05 Compare June 19, 2026 04:43

d4straub approved these changes Jun 19, 2026

View reviewed changes

dialvarezs added 2 commits June 19, 2026 10:43

chore: reorganize polishing code to make it more clear

52b1972

chore: add pypolca to output docs

a622a33

dialvarezs approved these changes Jun 19, 2026

View reviewed changes

Harshita-sriv merged commit e7d60f5 into nf-core:dev Jun 19, 2026
24 checks passed

Conversation

Harshita-sriv commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

dialvarezs commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dialvarezs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshita-sriv commented Jun 2, 2026

Uh oh!

dialvarezs commented Jun 4, 2026

Uh oh!

d4straub left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshita-sriv commented Jun 10, 2026

Uh oh!

d4straub commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Harshita-sriv commented Jun 12, 2026

Uh oh!

d4straub commented Jun 12, 2026

Uh oh!

dialvarezs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Harshita-sriv commented Jun 16, 2026

Uh oh!

dialvarezs commented Jun 16, 2026

Uh oh!

dialvarezs commented Jun 16, 2026

Uh oh!

Harshita-sriv commented Jun 16, 2026

Uh oh!

d4straub left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshita-sriv commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

d4straub Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Harshita-sriv Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dialvarezs Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dialvarezs Jun 18, 2026

Choose a reason for hiding this comment

Uh oh!

Harshita-sriv Jun 19, 2026

Choose a reason for hiding this comment

Uh oh!

d4straub left a comment

Choose a reason for hiding this comment

Uh oh!

dialvarezs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Harshita-sriv commented Jun 2, 2026 •

edited

Loading

dialvarezs commented Jun 2, 2026 •

edited

Loading

dialvarezs left a comment •

edited

Loading

d4straub commented Jun 11, 2026 •

edited

Loading

d4straub left a comment •

edited

Loading

Harshita-sriv commented Jun 18, 2026 •

edited

Loading

dialvarezs Jun 18, 2026 •

edited

Loading

dialvarezs left a comment •

edited

Loading