Repository navigation
Tags
Tags
Release PR (#115) * Skip fastqc template feature * Fix .nf-core.yml * Fix pipeline initialization * Init top-level subworkflow structure * Fix top-level structure bugs * Strength samplesheet validation * Add tool selection parameters * Pass methods to subworkflows * Update nf-core modules and subworkflows * Add method condition blocks * Install existing nf-core modules * Simplify raw/filtered matrix treatment * Improve hash/genetic channel structures * Patch demuxem module * Implement demuxem * Add GMM-Demux * Update genetic parameterization * Add POPSCLE_DSCPILEUP * Implement demuxlet * Implement freemuxlet * Implement cellsnp * Implement vireo * Implement BAM QC * Use PLP for demuxlet * Implement BAM subsetting to common variants * Update outdir structure * Update demuxem output definition * Add all demuxem optional parameters * Add GMM-demux optional params * Add cellSNP optional parameters * Add vireo optional params * Add popscle-dscpileup optional parameters * Add demuxlet/freemuxlet optional params * Use github CI runners * Add new hash demultiplexing modules (#66) * hasheddrops inserted and they run in the pipeline * insert local preprocessing module for htodemux and multiseq (not tested yet) * adding htodemux and multiseq to the pipeline * small improvements * add parameters of modules also to the pipeline * address * run pre-commit for modules/local/preprocessing_for_htodemux_multiseq * add htodemux visualization (not tested yet) * visualization added to the pipeline (works locally) * add visualization params to config files * change strings to type boolean for htodemux visualization * improve visualization * change data types for preprocessing * change data types of nf-core modules if possible * Prettier * Pre-commit * Fix nf-core modules not matching remote * Fix PULL_REQUEST_TEMPLATE issue based on this thread: https://nfcore.slack.com/archives/CQY2U5QU9/p1747475193145499 * Solve remaining prettier issues * One more prettier fix :D * Revert "Fix nf-core modules not matching remote" This reverts commit ac3ed24. * Make test profile work * Patch modules * Prettier * Gitignore test intermediates * Add pipeline test snapshot * Make sure hash diffs are visible * Ignore some more test snaps * Enable new hash-based tools * Set prefix to prevent file name collisions * nft-ignore some outputs --------- Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Aggregate hashing (#68) * first steps * current changes (not tested yet) * save changes * htodemux and multiseq in hash_summary.py (without mudata and anndata) and debugging of ch_hashing_summary * solve summary input with join * pipeline finally calls the script * empty inputs solved * add all inputs to main.nf (empty inputs still work) * update htodemux in script * script passed and produced results * correct results * solve incorrect merging * add demuxem (runs but merging to the summary table is not working) * remove the duplicate variable name * add demuxem to summary (has more barcodes than htodemux and multiseq) * add hasheddrops * add gmmdemux to summary * first steps to improve hasheddrops outputs * fix hasheddrops output * fix using unknown as value * remove comments * adding hashsolo doesn't work (cannot access the correct cell_hashing_columns) * add bff summary function * bff works * join into barcodes from raw hto file to adress demuxem having more output barcodes that input barcodes * pipeline pass with class Arguments * move testing to Argument class * restructure hash_summary.py * mudata and anndata outputs are created * hashsolo works * replace manual str with args variable * update hasheddrops * update modules and improve linting * Skip “prettier” failing – covered in #69 * clean up code 1 * clean code 2 (remove hard-coded parameters) * clean code 3 (hash_sumary.py) * update modules bff and hashsolo * update schema * remove properties created by nf-core pipelines schema build --no-prompts * structure hashing paramters of nextflow.config * update hasheddrops (removed one TODO in hash summary, hardcoding of mapping from id to hash) * remove hardcoded hash_list * remove linting error * use left join for classification and small improvements for summary csv * remove debugging outputs * Fix demuxem MPLCONFIGDIR * Add hash_summary environment variables * Comment out arg printing in hash_summary * Update demuxem patch file * Prettier * update snapshot --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Template update to 3.3.2 (#74) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Update multiQC module * Fix vireo_cell_range issue * Template update to 3.4.1 (#79) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Fix some linting problems * Use github runners instead of self-hosted Updated runner configuration to use 'ubuntu-latest' for nf-test jobs. * Fix runner configuration in nf-test workflow * Template update for nf-core/tools version 3.4.1 * Fix logo images * Integration of genetic modules and donor matching (#78) * save first steps * join outputs * gene_summary merges results for vireo, demuxlet and freemuxlet * merge on Barcode * all modules work for gene_summary but demuxlet has still empty values after merging * prepare donor matching * try understanding donor match * debugging donor match * donor_match produces something * donor_match produces something * donor_match seems to produce correct output files now, but it still needs to be tested with properly joined data * pipeline is running with new input data and donor_match produces correct output * first updates to souporcell * update souporcell * final versions.yml * save anndata/mudata, output before joining and left join to rna barcodes * couln't fix demuxEM error, opened an issue at developer's github * gene/hash_summary: save mudata/anndata, print results overview fuction and clean code * include all methods in donor_match and update bff() in hash_summary * improve the readability of hadge.nf * use fasta from iGenomes for souporcell but not fully tested (see #77) * remove prints and linting errors * update snapshot * Merge origin/dev * update souporcell * Fix image files * update snapshot * update snapshot 2 * fasta updates * update snapshot with 3 modules each (hash/gene) * update fasta description * update snapshot * replace set with list to keep the same order * snapshot from codespace * add souporcell outputs to .nftignore * Add trailing newline to .nftignore * fix end of file * add freemuxlet/*/*.clust1.vcf.gz to .nftignore * Incorporate review comments * update snapshot --------- Co-authored-by: nictru <nictru32@gmail.com> * Add find_variants, tests, and documentation (#96) * implementation of find_variants * subset gt_donors in mode 'donor_match' and update nextflow schema * clean find_variants * clean subset_gt_donors (update env) * clean donor_match * fix donor_match * add versions.yml (closes #87) * remove own TODOs and add error message for HTO names (closes #75, closes #73) * remove rename_genes_to_features (not used anymore) * add an own anndata/mudata module and remove it from summary modules (closes #83) * code and dependency cleanup of summary scripts * code and dependency cleanup of create_anndata_mudata.py * cleanup outdir * add a comment * closes #76 * add method overview output (closes #86) * add warning method mentioned in #77 * add test for genetic mode (see #95) * add tests for hashing and donor_match mode (see #95) * update nf-core modules vireo and souporcell (closes #94) * update input paths and add new tests to the github workflow (closes #95) * remove duplicate parameter * fix linting errors * update snapshots for rescue and donor_match * codespace snapshot default.nf.test * fix create anndata/mudata issue (hto==null in genetic mode) * add contributors * change defaultBranch * update contributor information * add another contributor * Add intro page * add rescue and genetic mode to the documentation * update samplesheet and add usage documentation * restructure steps of the pipeline * outputs for hashing, genetic and summary * close details * place output files below the headline * update and describe donor_match and find_variant outputs * push pipeline png * add citation of the relevant tools and adjust outdir * add seo's orcid * move checking the params and samplesheet inputs to 'utils_nfcore_hadge_pipeline' * move the validation of hto_names to utils_nfcore_hadge_pipeline with a function * add ext.prefix of JOIN_RESULTS with regex * add specific profiles to the tests and test vcf files * add nft-vcf plugin * update snapshot for rescue and donor match mode * update profile name of donor_match * update default snap with codespaces * ingore cluster_genotypes.vcf, sort AnnData columns and mention test profiles in docs * bash script to update all test snapshots * update snapshots with codespaces * update snapshot from test_genetic * update ignore souporcell vcf * add a seed for reproducible t-SNE and PCA * add extract hashes env.yml * nf-core pipelines lint --fix rocrate_readme_sync * nf-core pipelines lint --dir . --fix files_unchanged * Revert "nf-core pipelines lint --dir . --fix files_unchanged" This reverts commit 9bdae0b. * add comment for ignoring cluster_genotypes.vcf * Samplesheet Input Requirements by Module * small fix in docs * remove rna_matrix from genetic mode as it is not needed * simplify preprocessing channels * clean up formatting * remove barcodes from donor_match because it was already joined on barcodes.tsv in gene_summary.py * simplify tuple nesting issue in ch_donor_match * update docs * test new snapshots * update donor_match snapshot * fix donor_match snapshot issue * test new jpeg image * remove the random seed from htodemux (snapshot differences are due to OS variations) * update snapshot hashing * update rescue snapshot * update genetic snapshot * only update *_tSNE_htodemux.jpeg snapshot and date (multiqc throws an error in codespaces) * incorporate feedback * nf-core pipelines lint --fix rocrate_readme_sync * own line for each sentence * Template update 3.5.2 (#103) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.5.2 * Update multiqc * Test with larger nf-core runners (#101) * add larger nf-core runners * add zenodo and citation * update citation * add missing bam files to docs * add random comment to trigger CI tests * remove nf-core TODO's * nf-core pipelines lint --fix rocrate_readme_sync * use BibTeX for citations * try another style for citations * try another style for citations (pre-commit) * change dropdown syntax to html * adjust syntax * add detailed docs for demuxlet * update format * update format 2 * Fix Numba caching error in container by setting NUMBA_CACHE_DIR * setup test_full * fix hash modules for full_test compatibility * remove demuxem TODOs * update tests and snapshots * current test setup * reduce computing/storage resources of test and test_full * closes #107 * update nextflow_schema.json * update docs * update test_full input * nf-core pipelines lint --fix rocrate_readme_sync * update snapshots * remove outdir from config files * update snapshots * increase volume of self-hosted runners * Remove aws.client.anonymous from global nextflow.config in concordance with template * fine tune vireo configurations * increase runner volume and update test snapshot with all tools * set plp as path input * use plp dir as input to fix aws error * new try * new try 2 * remove time constraint for souporcell * increase time for souporcell * update readme --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Important! Template update for nf-core/tools v4.0.2 (#109) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Fix nextflow linting errors * Update multiQC module * Fix nextflow lint warnings and remove redundant samplesheet input parameter. Use explicit closure parameters, prefix unused bindings, and drop dead HTODEMUX channel assignments. PIPELINE_INITIALISATION reads params.input directly, so the take input is no longer needed. * Use elvis operator for simplification * Update container config files --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> * Update modules and use topic channels for version capture where applicable (#110) * Update nf-core modules and fix bam_qc for new samtools API. Refresh installed modules from nf-core/modules, adapt BAM_QC to the updated samtools view/sort inputs, and regenerate module patch files. Drop obsolete hasheddrops and multiseqdemux patches that no longer differ from upstream. * Drop manual version mixing for nf-core topic outputs. Rely on the versions topic collected before MultiQC instead of passing samtools, umitools, untar, csvtk, bff, and hasheddrops versions through subworkflows. * Emit local module versions via topic and keep patched nf-core mixing. Local modules publish to the versions topic so subworkflows no longer mix their outputs manually, while patched nf-core modules without topic support continue to use traditional ch_versions plumbing. * Restore Harshil alignment on local module emit statements. Re-align output emit columns after adding topic: versions during the topics migration. * Fix compatibility issues * Simplify hto null branching * Fix hto branch logic while preserving dev hash channel behaviour. Use a side-channel branch so ch_hto stays intact for joins, and mix is_null rows with EXTRACT_HASHES hashes rather than not_null hto paths. * Update nf-test snapshots for topic-based version collection. Refresh pipeline software versions YAML after local modules publish to the versions topic. * Ignore non-deterministic tSNE plots in nf-test snapshots. tSNE htodemux JPEGs vary across CI and local Docker runs; exclude them from stable-path md5 checks. * Fix BFF meta map types and record updated module SHA. (#111) Correct meta output types in bff/meta.yml after the nf-core module update and sync modules.json. * Remove vulnerable PR-comment artifact pattern (#113) Co-authored-by: nf-core-bot <core@nf-co.re> * Important! Template update for nf-core/tools v4.0.3 (#112) * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 3.3.2 * Template update for nf-core/tools version 4.0.0 * Template update for nf-core/tools version 4.0.2 * Template update for nf-core/tools version 4.0.3 * resolve missing merge conflicts * fix linting errors * update mulitqc * nf-core pipelines lint --dir /workspaces/hadge --fix container_configs * Fix conda container config file linting * Align linting workflow * use stable_name in test * use input instead of params.input --------- Co-authored-by: Nico Trummer <nictru32@gmail.com> Co-authored-by: LuisHeinzlmeier <luis.heinzlmeier@tum.de> * Prepare Release PR (#114) * bump-version * update changelog * resolve lint errors/warnings * udpate snapshots * update conda envs * bumped SHA to fix similarity issue in CI * update containers * add missing versions * exclude png for testing * remove png and jpeg from shapshots * update donor_match snapshot * update test * update remaining snapshots * fix test_hashing * fix test_genetic * use || instead of | * add meta.yml files to local modules using claude * update resource usage of VIREO and SOUPORCELL * restore apptainer in container directive of local modules * restore apptainer in container directive of patched nf-core modules * add stub and when to FILTER_BAM * first fixes * make vireo filtered variants optional for FIND_VARIANTS * only run cellSNP when vireo or FIND_VARIANTS uses its output * update docs * only allow one samplesheet row in donor_match mode * add checkIfExists * fine tune resources of vireo and souporcell * add mqc plot * update snapshots --------- Co-authored-by: nictru <nictru32@gmail.com> Co-authored-by: nf-core bot <infrastructure@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re>