Repository navigation
Integration of genetic modules and donor matching - #78
Conversation
…eeds to be tested with properly joined data
There was a problem hiding this comment.
This is not a full review but rather some notes on how to improve the ´fasta´ handling. The secret sauce here is that this does not override the params.fasta if it is already set directly. This is because in nextflow, parameter assignments directly in the pipeline code have lower priority than assignments via any type of configuration file.
I wrote these change suggestions without testing them, so they may contain typos or be incomplete, but I think you can get the idea
|
The test is currently failing only because of two I actually don’t need either of these files for the pipeline. I only included them because the documentation listed them as the main outputs. For hadge, I could simply remove them. If we want to keep them, I’d have to manually adjust the files after running What do you think, @nictru? |
|
In a case like this you can make use of the The tests did not fail in the souporcell PR because of the way the tests are implemented there. |
|
|
|
The pipeline passed all tests! ✅ |
|
Great! Will review in the evening |
Zethson
left a comment
There was a problem hiding this comment.
Great!
I'll leave the Nextflow bits to Nico.
nictru
left a comment
There was a problem hiding this comment.
Looks good in general, just a few small things and open TODO strings
|
I think it's fine to merge this now; I can do the template update in a separate PR later. Not sure when I will have time for this. Reminder: The fact that we are one version behind the latest nf-core template would only be displayed as a note if we were within the nf-core organization. We only get an error because the displaying of the note fails. |
* Skip fastqc template feature * Fix .nf-core.yml * Fix pipeline initialization * Init top-level subworkflow structure * Fix top-level structure bugs * Strength samplesheet validation * Add tool selection parameters * Pass methods to subworkflows * Update nf-core modules and subworkflows * Add method condition blocks * Install existing nf-core modules * Simplify raw/filtered matrix treatment * Improve hash/genetic channel structures * Patch demuxem module * Implement demuxem * Add GMM-Demux * Update genetic parameterization * Add POPSCLE_DSCPILEUP * Implement demuxlet * Implement freemuxlet * Implement cellsnp * Implement vireo * Implement BAM QC * Use PLP for demuxlet * Implement BAM subsetting to common variants * Update outdir structure * Update demuxem output definition * Add all demuxem optional parameters * Add GMM-demux optional params * Add cellSNP optional parameters * Add vireo optional params * Add popscle-dscpileup optional parameters * Add demuxlet/freemuxlet optional params * Use github CI runners * Add new hash demultiplexing modules (#66) * hasheddrops inserted and they run in the pipeline * insert local preprocessing module for htodemux and multiseq (not tested yet) * adding htodemux and multiseq to the pipeline * small improvements * add parameters of modules also to the pipeline * address * run pre-commit for modules/local/preprocessing_for_htodemux_multiseq * add htodemux visualization (not tested yet) * visualization added to the pipeline (works locally) * add visualization params to config files * change strings to type boolean for htodemux visualization * improve visualization * change data types for preprocessing * change data types of nf-core modules if possible * Prettier * Pre-commit * Fix nf-core modules not matching remote * Fix PULL_REQUEST_TEMPLATE issue based on this thread: https://nfcore.slack.com/archives/CQY2U5QU9/p1747475193145499 * Solve remaining prettier issues * One more prettier fix :D * Revert "Fix nf-core modules not matching remote" This reverts commit ac3ed24. * Make test profile work * Patch modules * Prettier * Gitignore test intermediates * Add pipeline test snapshot * Make sure hash diffs are visible * Ignore some more test snaps * Enable new hash-based tools * Set prefix to prevent file name collisions * nft-ignore some outputs --------- Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Aggregate hashing (#68) * first steps * current changes (not tested yet) * save changes * htodemux and multiseq in hash_summary.py (without mudata and anndata) and debugging of ch_hashing_summary * solve summary input with join * pipeline finally calls the script * empty inputs solved * add all inputs to main.nf (empty inputs still work) * update htodemux in script * script passed and produced results * correct results * solve incorrect merging * add demuxem (runs but merging to the summary table is not working) * remove the duplicate variable name * add demuxem to summary (has more barcodes than htodemux and multiseq) * add hasheddrops * add gmmdemux to summary * first steps to improve hasheddrops outputs * fix hasheddrops output * fix using unknown as value * remove comments * adding hashsolo doesn't work (cannot access the correct cell_hashing_columns) * add bff summary function * bff works * join into barcodes from raw hto file to adress demuxem having more output barcodes that input barcodes * pipeline pass with class Arguments * move testing to Argument class * restructure hash_summary.py * mudata and anndata outputs are created * hashsolo works * replace manual str with args variable * update hasheddrops * update modules and improve linting * Skip “prettier” failing – covered in #69 * clean up code 1 * clean code 2 (remove hard-coded parameters) * clean code 3 (hash_sumary.py) * update modules bff and hashsolo * update schema * remove properties created by nf-core pipelines schema build --no-prompts * structure hashing paramters of nextflow.config * update hasheddrops (removed one TODO in hash summary, hardcoding of mapping from id to hash) * remove hardcoded hash_list * remove linting error * use left join for classification and small improvements for summary csv * remove debugging outputs * Fix demuxem MPLCONFIGDIR * Add hash_summary environment variables * Comment out arg printing in hash_summary * Update demuxem patch file * Prettier * update snapshot --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Template update to 3.3.2 (#74) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Update multiQC module * Fix vireo_cell_range issue * Template update to 3.4.1 (#79) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Fix some linting problems * Use github runners instead of self-hosted Updated runner configuration to use 'ubuntu-latest' for nf-test jobs. * Fix runner configuration in nf-test workflow * Template update for nf-core/tools version 3.4.1 * Fix logo images * Integration of genetic modules and donor matching (#78) * save first steps * join outputs * gene_summary merges results for vireo, demuxlet and freemuxlet * merge on Barcode * all modules work for gene_summary but demuxlet has still empty values after merging * prepare donor matching * try understanding donor match * debugging donor match * donor_match produces something * donor_match produces something * donor_match seems to produce correct output files now, but it still needs to be tested with properly joined data * pipeline is running with new input data and donor_match produces correct output * first updates to souporcell * update souporcell * final versions.yml * save anndata/mudata, output before joining and left join to rna barcodes * couln't fix demuxEM error, opened an issue at developer's github * gene/hash_summary: save mudata/anndata, print results overview fuction and clean code * include all methods in donor_match and update bff() in hash_summary * improve the readability of hadge.nf * use fasta from iGenomes for souporcell but not fully tested (see #77) * remove prints and linting errors * update snapshot * Merge origin/dev * update souporcell * Fix image files * update snapshot * update snapshot 2 * fasta updates * update snapshot with 3 modules each (hash/gene) * update fasta description * update snapshot * replace set with list to keep the same order * snapshot from codespace * add souporcell outputs to .nftignore * Add trailing newline to .nftignore * fix end of file * add freemuxlet/*/*.clust1.vcf.gz to .nftignore * Incorporate review comments * update snapshot --------- Co-authored-by: nictru <nictru32@gmail.com> * Add find_variants, tests, and documentation (#96) * implementation of find_variants * subset gt_donors in mode 'donor_match' and update nextflow schema * clean find_variants * clean subset_gt_donors (update env) * clean donor_match * fix donor_match * add versions.yml (closes #87) * remove own TODOs and add error message for HTO names (closes #75, closes #73) * remove rename_genes_to_features (not used anymore) * add an own anndata/mudata module and remove it from summary modules (closes #83) * code and dependency cleanup of summary scripts * code and dependency cleanup of create_anndata_mudata.py * cleanup outdir * add a comment * closes #76 * add method overview output (closes #86) * add warning method mentioned in #77 * add test for genetic mode (see #95) * add tests for hashing and donor_match mode (see #95) * update nf-core modules vireo and souporcell (closes #94) * update input paths and add new tests to the github workflow (closes #95) * remove duplicate parameter * fix linting errors * update snapshots for rescue and donor_match * codespace snapshot default.nf.test * fix create anndata/mudata issue (hto==null in genetic mode) * add contributors * change defaultBranch * update contributor information * add another contributor * Add intro page * add rescue and genetic mode to the documentation * update samplesheet and add usage documentation * restructure steps of the pipeline * outputs for hashing, genetic and summary * close details * place output files below the headline * update and describe donor_match and find_variant outputs * push pipeline png * add citation of the relevant tools and adjust outdir * add seo's orcid * move checking the params and samplesheet inputs to 'utils_nfcore_hadge_pipeline' * move the validation of hto_names to utils_nfcore_hadge_pipeline with a function * add ext.prefix of JOIN_RESULTS with regex * add specific profiles to the tests and test vcf files * add nft-vcf plugin * update snapshot for rescue and donor match mode * update profile name of donor_match * update default snap with codespaces * ingore cluster_genotypes.vcf, sort AnnData columns and mention test profiles in docs * bash script to update all test snapshots * update snapshots with codespaces * update snapshot from test_genetic * update ignore souporcell vcf * add a seed for reproducible t-SNE and PCA * add extract hashes env.yml * nf-core pipelines lint --fix rocrate_readme_sync * nf-core pipelines lint --dir . --fix files_unchanged * Revert "nf-core pipelines lint --dir . --fix files_unchanged" This reverts commit 9bdae0b. * add comment for ignoring cluster_genotypes.vcf * Samplesheet Input Requirements by Module * small fix in docs * remove rna_matrix from genetic mode as it is not needed * simplify preprocessing channels * clean up formatting * remove barcodes from donor_match because it was already joined on barcodes.tsv in gene_summary.py * simplify tuple nesting issue in ch_donor_match * update docs * test new snapshots * update donor_match snapshot * fix donor_match snapshot issue * test new jpeg image * remove the random seed from htodemux (snapshot differences are due to OS variations) * update snapshot hashing * update rescue snapshot * update genetic snapshot * only update *_tSNE_htodemux.jpeg snapshot and date (multiqc throws an error in codespaces) * incorporate feedback * nf-core pipelines lint --fix rocrate_readme_sync * own line for each sentence * Template update 3.5.2 (#103) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.5.2 * Update multiqc * Test with larger nf-core runners (#101) * add larger nf-core runners * add zenodo and citation * update citation * add missing bam files to docs * add random comment to trigger CI tests * remove nf-core TODO's * nf-core pipelines lint --fix rocrate_readme_sync * use BibTeX for citations * try another style for citations * try another style for citations (pre-commit) * change dropdown syntax to html * adjust syntax * add detailed docs for demuxlet * update format * update format 2 * Fix Numba caching error in container by setting NUMBA_CACHE_DIR * setup test_full * fix hash modules for full_test compatibility * remove demuxem TODOs * update tests and snapshots * current test setup * reduce computing/storage resources of test and test_full * closes #107 * update nextflow_schema.json * update docs * update test_full input * nf-core pipelines lint --fix rocrate_readme_sync * update snapshots * remove outdir from config files * update snapshots * increase volume of self-hosted runners * Remove aws.client.anonymous from global nextflow.config in concordance with template * fine tune vireo configurations * increase runner volume and update test snapshot with all tools * set plp as path input * use plp dir as input to fix aws error * new try * new try 2 * remove time constraint for souporcell * increase time for souporcell * update readme --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Important! Template update for nf-core/tools v4.0.2 (#109) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Fix nextflow linting errors * Update multiQC module * Fix nextflow lint warnings and remove redundant samplesheet input parameter. Use explicit closure parameters, prefix unused bindings, and drop dead HTODEMUX channel assignments. PIPELINE_INITIALISATION reads params.input directly, so the take input is no longer needed. * Use elvis operator for simplification * Update container config files --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Update modules and use topic channels for version capture where applicable (#110) * Update nf-core modules and fix bam_qc for new samtools API. Refresh installed modules from nf-core/modules, adapt BAM_QC to the updated samtools view/sort inputs, and regenerate module patch files. Drop obsolete hasheddrops and multiseqdemux patches that no longer differ from upstream. * Drop manual version mixing for nf-core topic outputs. Rely on the versions topic collected before MultiQC instead of passing samtools, umitools, untar, csvtk, bff, and hasheddrops versions through subworkflows. * Emit local module versions via topic and keep patched nf-core mixing. Local modules publish to the versions topic so subworkflows no longer mix their outputs manually, while patched nf-core modules without topic support continue to use traditional ch_versions plumbing. * Restore Harshil alignment on local module emit statements. Re-align output emit columns after adding topic: versions during the topics migration. * Fix compatibility issues * Simplify hto null branching * Fix hto branch logic while preserving dev hash channel behaviour. Use a side-channel branch so ch_hto stays intact for joins, and mix is_null rows with EXTRACT_HASHES hashes rather than not_null hto paths. * Update nf-test snapshots for topic-based version collection. Refresh pipeline software versions YAML after local modules publish to the versions topic. * Ignore non-deterministic tSNE plots in nf-test snapshots. tSNE htodemux JPEGs vary across CI and local Docker runs; exclude them from stable-path md5 checks. * Fix BFF meta map types and record updated module SHA. (#111) Correct meta output types in bff/meta.yml after the nf-core module update and sync modules.json. * Remove vulnerable PR-comment artifact pattern (#113) Co-authored-by: nf-core-bot <core@nf-co.re> * Important! Template update for nf-core/tools v4.0.3 (#112) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Template update for nf-core/tools version 4.0.3 * resolve missing merge conflicts * fix linting errors * update mulitqc * nf-core pipelines lint --dir /workspaces/hadge --fix container_configs * Fix conda container config file linting * Align linting workflow * use stable_name in test * use input instead of params.input --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Prepare Release PR (#114) * bump-version * update changelog * resolve lint errors/warnings * udpate snapshots * update conda envs * bumped SHA to fix similarity issue in CI * update containers * add missing versions * exclude png for testing * remove png and jpeg from shapshots * update donor_match snapshot * update test * update remaining snapshots * fix test_hashing * fix test_genetic * use || instead of | * add meta.yml files to local modules using claude * update resource usage of VIREO and SOUPORCELL * restore apptainer in container directive of local modules * restore apptainer in container directive of patched nf-core modules * add stub and when to FILTER_BAM * first fixes * make vireo filtered variants optional for FIND_VARIANTS * only run cellSNP when vireo or FIND_VARIANTS uses its output * update docs * only allow one samplesheet row in donor_match mode * add checkIfExists * fine tune resources of vireo and souporcell * add mqc plot * update snapshots --------- Co-authored-by: nictru <nictru32@gmail.com> Co-authored-by: nf-core bot <infrastructure@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re>
* Skip fastqc template feature * Fix .nf-core.yml * Fix pipeline initialization * Init top-level subworkflow structure * Fix top-level structure bugs * Strength samplesheet validation * Add tool selection parameters * Pass methods to subworkflows * Update nf-core modules and subworkflows * Add method condition blocks * Install existing nf-core modules * Simplify raw/filtered matrix treatment * Improve hash/genetic channel structures * Patch demuxem module * Implement demuxem * Add GMM-Demux * Update genetic parameterization * Add POPSCLE_DSCPILEUP * Implement demuxlet * Implement freemuxlet * Implement cellsnp * Implement vireo * Implement BAM QC * Use PLP for demuxlet * Implement BAM subsetting to common variants * Update outdir structure * Update demuxem output definition * Add all demuxem optional parameters * Add GMM-demux optional params * Add cellSNP optional parameters * Add vireo optional params * Add popscle-dscpileup optional parameters * Add demuxlet/freemuxlet optional params * Use github CI runners * Add new hash demultiplexing modules (#66) * hasheddrops inserted and they run in the pipeline * insert local preprocessing module for htodemux and multiseq (not tested yet) * adding htodemux and multiseq to the pipeline * small improvements * add parameters of modules also to the pipeline * address * run pre-commit for modules/local/preprocessing_for_htodemux_multiseq * add htodemux visualization (not tested yet) * visualization added to the pipeline (works locally) * add visualization params to config files * change strings to type boolean for htodemux visualization * improve visualization * change data types for preprocessing * change data types of nf-core modules if possible * Prettier * Pre-commit * Fix nf-core modules not matching remote * Fix PULL_REQUEST_TEMPLATE issue based on this thread: https://nfcore.slack.com/archives/CQY2U5QU9/p1747475193145499 * Solve remaining prettier issues * One more prettier fix :D * Revert "Fix nf-core modules not matching remote" This reverts commit ac3ed24. * Make test profile work * Patch modules * Prettier * Gitignore test intermediates * Add pipeline test snapshot * Make sure hash diffs are visible * Ignore some more test snaps * Enable new hash-based tools * Set prefix to prevent file name collisions * nft-ignore some outputs --------- Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Aggregate hashing (#68) * first steps * current changes (not tested yet) * save changes * htodemux and multiseq in hash_summary.py (without mudata and anndata) and debugging of ch_hashing_summary * solve summary input with join * pipeline finally calls the script * empty inputs solved * add all inputs to main.nf (empty inputs still work) * update htodemux in script * script passed and produced results * correct results * solve incorrect merging * add demuxem (runs but merging to the summary table is not working) * remove the duplicate variable name * add demuxem to summary (has more barcodes than htodemux and multiseq) * add hasheddrops * add gmmdemux to summary * first steps to improve hasheddrops outputs * fix hasheddrops output * fix using unknown as value * remove comments * adding hashsolo doesn't work (cannot access the correct cell_hashing_columns) * add bff summary function * bff works * join into barcodes from raw hto file to adress demuxem having more output barcodes that input barcodes * pipeline pass with class Arguments * move testing to Argument class * restructure hash_summary.py * mudata and anndata outputs are created * hashsolo works * replace manual str with args variable * update hasheddrops * update modules and improve linting * Skip “prettier” failing – covered in #69 * clean up code 1 * clean code 2 (remove hard-coded parameters) * clean code 3 (hash_sumary.py) * update modules bff and hashsolo * update schema * remove properties created by nf-core pipelines schema build --no-prompts * structure hashing paramters of nextflow.config * update hasheddrops (removed one TODO in hash summary, hardcoding of mapping from id to hash) * remove hardcoded hash_list * remove linting error * use left join for classification and small improvements for summary csv * remove debugging outputs * Fix demuxem MPLCONFIGDIR * Add hash_summary environment variables * Comment out arg printing in hash_summary * Update demuxem patch file * Prettier * update snapshot --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Template update to 3.3.2 (#74) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Update multiQC module * Fix vireo_cell_range issue * Template update to 3.4.1 (#79) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Fix some linting problems * Use github runners instead of self-hosted Updated runner configuration to use 'ubuntu-latest' for nf-test jobs. * Fix runner configuration in nf-test workflow * Template update for nf-core/tools version 3.4.1 * Fix logo images * Integration of genetic modules and donor matching (#78) * save first steps * join outputs * gene_summary merges results for vireo, demuxlet and freemuxlet * merge on Barcode * all modules work for gene_summary but demuxlet has still empty values after merging * prepare donor matching * try understanding donor match * debugging donor match * donor_match produces something * donor_match produces something * donor_match seems to produce correct output files now, but it still needs to be tested with properly joined data * pipeline is running with new input data and donor_match produces correct output * first updates to souporcell * update souporcell * final versions.yml * save anndata/mudata, output before joining and left join to rna barcodes * couln't fix demuxEM error, opened an issue at developer's github * gene/hash_summary: save mudata/anndata, print results overview fuction and clean code * include all methods in donor_match and update bff() in hash_summary * improve the readability of hadge.nf * use fasta from iGenomes for souporcell but not fully tested (see #77) * remove prints and linting errors * update snapshot * Merge origin/dev * update souporcell * Fix image files * update snapshot * update snapshot 2 * fasta updates * update snapshot with 3 modules each (hash/gene) * update fasta description * update snapshot * replace set with list to keep the same order * snapshot from codespace * add souporcell outputs to .nftignore * Add trailing newline to .nftignore * fix end of file * add freemuxlet/*/*.clust1.vcf.gz to .nftignore * Incorporate review comments * update snapshot --------- Co-authored-by: nictru <nictru32@gmail.com> * Add find_variants, tests, and documentation (#96) * implementation of find_variants * subset gt_donors in mode 'donor_match' and update nextflow schema * clean find_variants * clean subset_gt_donors (update env) * clean donor_match * fix donor_match * add versions.yml (closes #87) * remove own TODOs and add error message for HTO names (closes #75, closes #73) * remove rename_genes_to_features (not used anymore) * add an own anndata/mudata module and remove it from summary modules (closes #83) * code and dependency cleanup of summary scripts * code and dependency cleanup of create_anndata_mudata.py * cleanup outdir * add a comment * closes #76 * add method overview output (closes #86) * add warning method mentioned in #77 * add test for genetic mode (see #95) * add tests for hashing and donor_match mode (see #95) * update nf-core modules vireo and souporcell (closes #94) * update input paths and add new tests to the github workflow (closes #95) * remove duplicate parameter * fix linting errors * update snapshots for rescue and donor_match * codespace snapshot default.nf.test * fix create anndata/mudata issue (hto==null in genetic mode) * add contributors * change defaultBranch * update contributor information * add another contributor * Add intro page * add rescue and genetic mode to the documentation * update samplesheet and add usage documentation * restructure steps of the pipeline * outputs for hashing, genetic and summary * close details * place output files below the headline * update and describe donor_match and find_variant outputs * push pipeline png * add citation of the relevant tools and adjust outdir * add seo's orcid * move checking the params and samplesheet inputs to 'utils_nfcore_hadge_pipeline' * move the validation of hto_names to utils_nfcore_hadge_pipeline with a function * add ext.prefix of JOIN_RESULTS with regex * add specific profiles to the tests and test vcf files * add nft-vcf plugin * update snapshot for rescue and donor match mode * update profile name of donor_match * update default snap with codespaces * ingore cluster_genotypes.vcf, sort AnnData columns and mention test profiles in docs * bash script to update all test snapshots * update snapshots with codespaces * update snapshot from test_genetic * update ignore souporcell vcf * add a seed for reproducible t-SNE and PCA * add extract hashes env.yml * nf-core pipelines lint --fix rocrate_readme_sync * nf-core pipelines lint --dir . --fix files_unchanged * Revert "nf-core pipelines lint --dir . --fix files_unchanged" This reverts commit 9bdae0b. * add comment for ignoring cluster_genotypes.vcf * Samplesheet Input Requirements by Module * small fix in docs * remove rna_matrix from genetic mode as it is not needed * simplify preprocessing channels * clean up formatting * remove barcodes from donor_match because it was already joined on barcodes.tsv in gene_summary.py * simplify tuple nesting issue in ch_donor_match * update docs * test new snapshots * update donor_match snapshot * fix donor_match snapshot issue * test new jpeg image * remove the random seed from htodemux (snapshot differences are due to OS variations) * update snapshot hashing * update rescue snapshot * update genetic snapshot * only update *_tSNE_htodemux.jpeg snapshot and date (multiqc throws an error in codespaces) * incorporate feedback * nf-core pipelines lint --fix rocrate_readme_sync * own line for each sentence * Template update 3.5.2 (#103) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.5.2 * Update multiqc * Test with larger nf-core runners (#101) * add larger nf-core runners * add zenodo and citation * update citation * add missing bam files to docs * add random comment to trigger CI tests * remove nf-core TODO's * nf-core pipelines lint --fix rocrate_readme_sync * use BibTeX for citations * try another style for citations * try another style for citations (pre-commit) * change dropdown syntax to html * adjust syntax * add detailed docs for demuxlet * update format * update format 2 * Fix Numba caching error in container by setting NUMBA_CACHE_DIR * setup test_full * fix hash modules for full_test compatibility * remove demuxem TODOs * update tests and snapshots * current test setup * reduce computing/storage resources of test and test_full * closes #107 * update nextflow_schema.json * update docs * update test_full input * nf-core pipelines lint --fix rocrate_readme_sync * update snapshots * remove outdir from config files * update snapshots * increase volume of self-hosted runners * Remove aws.client.anonymous from global nextflow.config in concordance with template * fine tune vireo configurations * increase runner volume and update test snapshot with all tools * set plp as path input * use plp dir as input to fix aws error * new try * new try 2 * remove time constraint for souporcell * increase time for souporcell * update readme --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Important! Template update for nf-core/tools v4.0.2 (#109) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Fix nextflow linting errors * Update multiQC module * Fix nextflow lint warnings and remove redundant samplesheet input parameter. Use explicit closure parameters, prefix unused bindings, and drop dead HTODEMUX channel assignments. PIPELINE_INITIALISATION reads params.input directly, so the take input is no longer needed. * Use elvis operator for simplification * Update container config files --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Update modules and use topic channels for version capture where applicable (#110) * Update nf-core modules and fix bam_qc for new samtools API. Refresh installed modules from nf-core/modules, adapt BAM_QC to the updated samtools view/sort inputs, and regenerate module patch files. Drop obsolete hasheddrops and multiseqdemux patches that no longer differ from upstream. * Drop manual version mixing for nf-core topic outputs. Rely on the versions topic collected before MultiQC instead of passing samtools, umitools, untar, csvtk, bff, and hasheddrops versions through subworkflows. * Emit local module versions via topic and keep patched nf-core mixing. Local modules publish to the versions topic so subworkflows no longer mix their outputs manually, while patched nf-core modules without topic support continue to use traditional ch_versions plumbing. * Restore Harshil alignment on local module emit statements. Re-align output emit columns after adding topic: versions during the topics migration. * Fix compatibility issues * Simplify hto null branching * Fix hto branch logic while preserving dev hash channel behaviour. Use a side-channel branch so ch_hto stays intact for joins, and mix is_null rows with EXTRACT_HASHES hashes rather than not_null hto paths. * Update nf-test snapshots for topic-based version collection. Refresh pipeline software versions YAML after local modules publish to the versions topic. * Ignore non-deterministic tSNE plots in nf-test snapshots. tSNE htodemux JPEGs vary across CI and local Docker runs; exclude them from stable-path md5 checks. * Fix BFF meta map types and record updated module SHA. (#111) Correct meta output types in bff/meta.yml after the nf-core module update and sync modules.json. * Remove vulnerable PR-comment artifact pattern (#113) Co-authored-by: nf-core-bot <core@nf-co.re> * Important! Template update for nf-core/tools v4.0.3 (#112) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Template update for nf-core/tools version 4.0.3 * resolve missing merge conflicts * fix linting errors * update mulitqc * nf-core pipelines lint --dir /workspaces/hadge --fix container_configs * Fix conda container config file linting * Align linting workflow * use stable_name in test * use input instead of params.input --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Prepare Release PR (#114) * bump-version * update changelog * resolve lint errors/warnings * udpate snapshots * update conda envs * bumped SHA to fix similarity issue in CI * update containers * add missing versions * exclude png for testing * remove png and jpeg from shapshots * update donor_match snapshot * update test * update remaining snapshots * fix test_hashing * fix test_genetic * use || instead of | * add meta.yml files to local modules using claude * update resource usage of VIREO and SOUPORCELL * restore apptainer in container directive of local modules * restore apptainer in container directive of patched nf-core modules * add stub and when to FILTER_BAM * first fixes * make vireo filtered variants optional for FIND_VARIANTS * only run cellSNP when vireo or FIND_VARIANTS uses its output * update docs * only allow one samplesheet row in donor_match mode * add checkIfExists * fine tune resources of vireo and souporcell * add mqc plot * update snapshots --------- Co-authored-by: nictru <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> Co-authored-by: nf-core bot <infrastructure@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re>
* Skip fastqc template feature * Fix .nf-core.yml * Fix pipeline initialization * Init top-level subworkflow structure * Fix top-level structure bugs * Strength samplesheet validation * Add tool selection parameters * Pass methods to subworkflows * Update nf-core modules and subworkflows * Add method condition blocks * Install existing nf-core modules * Simplify raw/filtered matrix treatment * Improve hash/genetic channel structures * Patch demuxem module * Implement demuxem * Add GMM-Demux * Update genetic parameterization * Add POPSCLE_DSCPILEUP * Implement demuxlet * Implement freemuxlet * Implement cellsnp * Implement vireo * Implement BAM QC * Use PLP for demuxlet * Implement BAM subsetting to common variants * Update outdir structure * Update demuxem output definition * Add all demuxem optional parameters * Add GMM-demux optional params * Add cellSNP optional parameters * Add vireo optional params * Add popscle-dscpileup optional parameters * Add demuxlet/freemuxlet optional params * Use github CI runners * Add new hash demultiplexing modules (#66) * hasheddrops inserted and they run in the pipeline * insert local preprocessing module for htodemux and multiseq (not tested yet) * adding htodemux and multiseq to the pipeline * small improvements * add parameters of modules also to the pipeline * address * run pre-commit for modules/local/preprocessing_for_htodemux_multiseq * add htodemux visualization (not tested yet) * visualization added to the pipeline (works locally) * add visualization params to config files * change strings to type boolean for htodemux visualization * improve visualization * change data types for preprocessing * change data types of nf-core modules if possible * Prettier * Pre-commit * Fix nf-core modules not matching remote * Fix PULL_REQUEST_TEMPLATE issue based on this thread: https://nfcore.slack.com/archives/CQY2U5QU9/p1747475193145499 * Solve remaining prettier issues * One more prettier fix :D * Revert "Fix nf-core modules not matching remote" This reverts commit ac3ed24. * Make test profile work * Patch modules * Prettier * Gitignore test intermediates * Add pipeline test snapshot * Make sure hash diffs are visible * Ignore some more test snaps * Enable new hash-based tools * Set prefix to prevent file name collisions * nft-ignore some outputs --------- Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Aggregate hashing (#68) * first steps * current changes (not tested yet) * save changes * htodemux and multiseq in hash_summary.py (without mudata and anndata) and debugging of ch_hashing_summary * solve summary input with join * pipeline finally calls the script * empty inputs solved * add all inputs to main.nf (empty inputs still work) * update htodemux in script * script passed and produced results * correct results * solve incorrect merging * add demuxem (runs but merging to the summary table is not working) * remove the duplicate variable name * add demuxem to summary (has more barcodes than htodemux and multiseq) * add hasheddrops * add gmmdemux to summary * first steps to improve hasheddrops outputs * fix hasheddrops output * fix using unknown as value * remove comments * adding hashsolo doesn't work (cannot access the correct cell_hashing_columns) * add bff summary function * bff works * join into barcodes from raw hto file to adress demuxem having more output barcodes that input barcodes * pipeline pass with class Arguments * move testing to Argument class * restructure hash_summary.py * mudata and anndata outputs are created * hashsolo works * replace manual str with args variable * update hasheddrops * update modules and improve linting * Skip “prettier” failing – covered in #69 * clean up code 1 * clean code 2 (remove hard-coded parameters) * clean code 3 (hash_sumary.py) * update modules bff and hashsolo * update schema * remove properties created by nf-core pipelines schema build --no-prompts * structure hashing paramters of nextflow.config * update hasheddrops (removed one TODO in hash summary, hardcoding of mapping from id to hash) * remove hardcoded hash_list * remove linting error * use left join for classification and small improvements for summary csv * remove debugging outputs * Fix demuxem MPLCONFIGDIR * Add hash_summary environment variables * Comment out arg printing in hash_summary * Update demuxem patch file * Prettier * update snapshot --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Template update to 3.3.2 (#74) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Update multiQC module * Fix vireo_cell_range issue * Template update to 3.4.1 (#79) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Fix some linting problems * Use github runners instead of self-hosted Updated runner configuration to use 'ubuntu-latest' for nf-test jobs. * Fix runner configuration in nf-test workflow * Template update for nf-core/tools version 3.4.1 * Fix logo images * Integration of genetic modules and donor matching (#78) * save first steps * join outputs * gene_summary merges results for vireo, demuxlet and freemuxlet * merge on Barcode * all modules work for gene_summary but demuxlet has still empty values after merging * prepare donor matching * try understanding donor match * debugging donor match * donor_match produces something * donor_match produces something * donor_match seems to produce correct output files now, but it still needs to be tested with properly joined data * pipeline is running with new input data and donor_match produces correct output * first updates to souporcell * update souporcell * final versions.yml * save anndata/mudata, output before joining and left join to rna barcodes * couln't fix demuxEM error, opened an issue at developer's github * gene/hash_summary: save mudata/anndata, print results overview fuction and clean code * include all methods in donor_match and update bff() in hash_summary * improve the readability of hadge.nf * use fasta from iGenomes for souporcell but not fully tested (see #77) * remove prints and linting errors * update snapshot * Merge origin/dev * update souporcell * Fix image files * update snapshot * update snapshot 2 * fasta updates * update snapshot with 3 modules each (hash/gene) * update fasta description * update snapshot * replace set with list to keep the same order * snapshot from codespace * add souporcell outputs to .nftignore * Add trailing newline to .nftignore * fix end of file * add freemuxlet/*/*.clust1.vcf.gz to .nftignore * Incorporate review comments * update snapshot --------- Co-authored-by: nictru <nictru32@gmail.com> * Add find_variants, tests, and documentation (#96) * implementation of find_variants * subset gt_donors in mode 'donor_match' and update nextflow schema * clean find_variants * clean subset_gt_donors (update env) * clean donor_match * fix donor_match * add versions.yml (closes #87) * remove own TODOs and add error message for HTO names (closes #75, closes #73) * remove rename_genes_to_features (not used anymore) * add an own anndata/mudata module and remove it from summary modules (closes #83) * code and dependency cleanup of summary scripts * code and dependency cleanup of create_anndata_mudata.py * cleanup outdir * add a comment * closes #76 * add method overview output (closes #86) * add warning method mentioned in #77 * add test for genetic mode (see #95) * add tests for hashing and donor_match mode (see #95) * update nf-core modules vireo and souporcell (closes #94) * update input paths and add new tests to the github workflow (closes #95) * remove duplicate parameter * fix linting errors * update snapshots for rescue and donor_match * codespace snapshot default.nf.test * fix create anndata/mudata issue (hto==null in genetic mode) * add contributors * change defaultBranch * update contributor information * add another contributor * Add intro page * add rescue and genetic mode to the documentation * update samplesheet and add usage documentation * restructure steps of the pipeline * outputs for hashing, genetic and summary * close details * place output files below the headline * update and describe donor_match and find_variant outputs * push pipeline png * add citation of the relevant tools and adjust outdir * add seo's orcid * move checking the params and samplesheet inputs to 'utils_nfcore_hadge_pipeline' * move the validation of hto_names to utils_nfcore_hadge_pipeline with a function * add ext.prefix of JOIN_RESULTS with regex * add specific profiles to the tests and test vcf files * add nft-vcf plugin * update snapshot for rescue and donor match mode * update profile name of donor_match * update default snap with codespaces * ingore cluster_genotypes.vcf, sort AnnData columns and mention test profiles in docs * bash script to update all test snapshots * update snapshots with codespaces * update snapshot from test_genetic * update ignore souporcell vcf * add a seed for reproducible t-SNE and PCA * add extract hashes env.yml * nf-core pipelines lint --fix rocrate_readme_sync * nf-core pipelines lint --dir . --fix files_unchanged * Revert "nf-core pipelines lint --dir . --fix files_unchanged" This reverts commit 9bdae0b. * add comment for ignoring cluster_genotypes.vcf * Samplesheet Input Requirements by Module * small fix in docs * remove rna_matrix from genetic mode as it is not needed * simplify preprocessing channels * clean up formatting * remove barcodes from donor_match because it was already joined on barcodes.tsv in gene_summary.py * simplify tuple nesting issue in ch_donor_match * update docs * test new snapshots * update donor_match snapshot * fix donor_match snapshot issue * test new jpeg image * remove the random seed from htodemux (snapshot differences are due to OS variations) * update snapshot hashing * update rescue snapshot * update genetic snapshot * only update *_tSNE_htodemux.jpeg snapshot and date (multiqc throws an error in codespaces) * incorporate feedback * nf-core pipelines lint --fix rocrate_readme_sync * own line for each sentence * Template update 3.5.2 (#103) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.5.2 * Update multiqc * Test with larger nf-core runners (#101) * add larger nf-core runners * add zenodo and citation * update citation * add missing bam files to docs * add random comment to trigger CI tests * remove nf-core TODO's * nf-core pipelines lint --fix rocrate_readme_sync * use BibTeX for citations * try another style for citations * try another style for citations (pre-commit) * change dropdown syntax to html * adjust syntax * add detailed docs for demuxlet * update format * update format 2 * Fix Numba caching error in container by setting NUMBA_CACHE_DIR * setup test_full * fix hash modules for full_test compatibility * remove demuxem TODOs * update tests and snapshots * current test setup * reduce computing/storage resources of test and test_full * closes #107 * update nextflow_schema.json * update docs * update test_full input * nf-core pipelines lint --fix rocrate_readme_sync * update snapshots * remove outdir from config files * update snapshots * increase volume of self-hosted runners * Remove aws.client.anonymous from global nextflow.config in concordance with template * fine tune vireo configurations * increase runner volume and update test snapshot with all tools * set plp as path input * use plp dir as input to fix aws error * new try * new try 2 * remove time constraint for souporcell * increase time for souporcell * update readme --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Important! Template update for nf-core/tools v4.0.2 (#109) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Fix nextflow linting errors * Update multiQC module * Fix nextflow lint warnings and remove redundant samplesheet input parameter. Use explicit closure parameters, prefix unused bindings, and drop dead HTODEMUX channel assignments. PIPELINE_INITIALISATION reads params.input directly, so the take input is no longer needed. * Use elvis operator for simplification * Update container config files --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Update modules and use topic channels for version capture where applicable (#110) * Update nf-core modules and fix bam_qc for new samtools API. Refresh installed modules from nf-core/modules, adapt BAM_QC to the updated samtools view/sort inputs, and regenerate module patch files. Drop obsolete hasheddrops and multiseqdemux patches that no longer differ from upstream. * Drop manual version mixing for nf-core topic outputs. Rely on the versions topic collected before MultiQC instead of passing samtools, umitools, untar, csvtk, bff, and hasheddrops versions through subworkflows. * Emit local module versions via topic and keep patched nf-core mixing. Local modules publish to the versions topic so subworkflows no longer mix their outputs manually, while patched nf-core modules without topic support continue to use traditional ch_versions plumbing. * Restore Harshil alignment on local module emit statements. Re-align output emit columns after adding topic: versions during the topics migration. * Fix compatibility issues * Simplify hto null branching * Fix hto branch logic while preserving dev hash channel behaviour. Use a side-channel branch so ch_hto stays intact for joins, and mix is_null rows with EXTRACT_HASHES hashes rather than not_null hto paths. * Update nf-test snapshots for topic-based version collection. Refresh pipeline software versions YAML after local modules publish to the versions topic. * Ignore non-deterministic tSNE plots in nf-test snapshots. tSNE htodemux JPEGs vary across CI and local Docker runs; exclude them from stable-path md5 checks. * Fix BFF meta map types and record updated module SHA. (#111) Correct meta output types in bff/meta.yml after the nf-core module update and sync modules.json. * Remove vulnerable PR-comment artifact pattern (#113) Co-authored-by: nf-core-bot <core@nf-co.re> * Important! Template update for nf-core/tools v4.0.3 (#112) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Template update for nf-core/tools version 4.0.3 * resolve missing merge conflicts * fix linting errors * update mulitqc * nf-core pipelines lint --dir /workspaces/hadge --fix container_configs * Fix conda container config file linting * Align linting workflow * use stable_name in test * use input instead of params.input --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Prepare Release PR (#114) * bump-version * update changelog * resolve lint errors/warnings * udpate snapshots * update conda envs * bumped SHA to fix similarity issue in CI * update containers * add missing versions * exclude png for testing * remove png and jpeg from shapshots * update donor_match snapshot * update test * update remaining snapshots * fix test_hashing * fix test_genetic * use || instead of | * add meta.yml files to local modules using claude * update resource usage of VIREO and SOUPORCELL * restore apptainer in container directive of local modules * restore apptainer in container directive of patched nf-core modules * add stub and when to FILTER_BAM * first fixes * make vireo filtered variants optional for FIND_VARIANTS * only run cellSNP when vireo or FIND_VARIANTS uses its output * update docs * only allow one samplesheet row in donor_match mode * add checkIfExists * fine tune resources of vireo and souporcell * add mqc plot * update snapshots * feat: migrate to the nf-core/tools 4.1.0 template with nf-migrate Applies the migration chain from dev's template version to 4.1.0. The CHANGELOG.md and conf/modules.config changes did not apply and follow as separate commits. Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs * style: apply nf-core pipelines lint --fix * docs(changelog): add the 4.1.0 template's unreleased entry Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs * fix(config): drop saveAs from the default publishDir like template 4.1.0 Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs * chore(containers): regenerate the conda lock container configs Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs * chore(ro-crate): sync the RO-Crate description with the README Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs * fix(subworkflows): update utils_nfschema_plugin to the version the 4.1.0 template calls The template passes ten inputs (help_full was added); the installed version takes nine, so the pipeline failed to initialise. Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs - Fix the CI failure on the migration PR * fix(subworkflows): pin utils_nfschema_plugin to the 4.1.0 template's version The previous update installed a newer version that needs a newer nf-schema plugin than the one the template pins (2.5.1). Coding/Writing assisted with Claude Sonnet 5.5 Prompts: - Resolve nf-core/hadge PR 116 (template 4.1.0) with nf-migrate and open two stacked draft PRs - Fix the CI failure on the migration PR * fix(schema): point schema $id URLs at the main branch nf-migrate reintroduced the template's `master` default branch in the `$id` of nextflow_schema.json and assets/schema_input.json; this repo's default branch is `main` (manifest.defaultBranch). Coding/Writing assisted with Claude Opus 5.5 Prompts: - checkout #121 and fix comment - just push --------- Co-authored-by: nictru <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> Co-authored-by: nf-core bot <infrastructure@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re>
Updated dataset
demuxemis currently not working after this dataset update.--> demuxem does not work #81
Summarizing gene and hash results
souporcellat nf-core.--genome). However, this feature could not yet be fully tested.-->
souporcellfails with SIGKILL (OOM on laptop) #77--> Return a file instead of printing overview #86
Donor match
findVariant = truestill needs to be implemented.--> Implement findVariant in donor_match #82
Thanks to @seokim for the huge contribution to aggregating the genetic modules!