Host read removal with Bowtie 2 by skrakau · Pull Request #49 · nf-core/mag

skrakau · 2020-05-31T10:40:59Z

Here is a suggestion for host read removal using bowtie2. For the host reference sequence either iGenomes or a user specified Fasta reference file can be used.

I also added a MultiQC section for this to display which fraction of reads maps against the host reference and is thus filtered out using a custom content file as suggested by @ewels and @drpatelh (MultiQC/MultiQC#1199), instead of using the standard Bowtie 2 MultiQC format (with its multiple different mapping categories). Moreover, I separated the MultiQC FastQC for before and after preprocessing.

If using this bowtie2 based strategy would be fine for you, I can add a test.

PR checklist

This comment contains a description of changes (with reason)
If you've fixed a bug or added code that should be tested, add tests!
If necessary, also make a PR on the nf-core/mag branch on the nf-core/test-datasets repo
Ensure the test suite passes (nextflow run . -profile test,docker).
Make sure your code lints (nf-core lint .).
Documentation in docs is updated
CHANGELOG.md is updated
README.md is updated

Learn more about contributing: https://github.com/nf-core/mag/tree/master/.github/CONTRIBUTING.md

…sing.

…stom content file

d4straub · 2020-06-02T07:54:20Z

Bowtie2 host removal is fine.
A test and documentation would be great.

skrakau · 2020-06-02T09:21:05Z

Ok thanks, will do!

ewels · 2020-06-02T10:03:31Z

+        zcat ${reads[0]} | echo "Read pairs before removal: \$((`wc -l`/4))" >>${name}_remove_host.log
+        zcat ${name}_host_unmapped_1.fastq.gz | echo "Read pairs after removal: \$((`wc -l`/4))" >>${name}_remove_host.log


Is this just for debugging? I can't see where ${name}_remove_host.log is used elsewhere.

Might be a bit of an expensive operation if so, zcat on a big FastQ file can take quite a while. And I guess we get this info from your bowtie logs MultiQC module anyway?

True, I originally copied it from the remove_phix process, but it can be removed here.

HadrienG · 2020-06-03T06:31:59Z

+
+    script:
+    def sensitivity = params.host_removal_verysensitive ? "--very-sensitive" : "--sensitive"
+    if ( !params.single_end ) {


for avoiding to duplicate most of the process code (sin gle end and paired end), you can define an input parameter, like at https://github.com/nf-core/mag/blob/master/main.nf#L672

ok, but in this case there are multiple parts in the process code affected:

-1 "${reads[0]}" -2 "${reads[1]}"

--un-conc-gz ${name}_host_unmapped_%.fastq.gz \
--al-conc-gz ${name}_host_mapped_%.fastq.gz \

zcat ${name}_host_mapped_1.fastq.gz | awk '{if(NR%4==1) print substr(\$0, 2)}' > ${name}_host_mapped_1.read_ids.txt
zcat ${name}_host_mapped_2.fastq.gz | awk '{if(NR%4==1) print substr(\$0, 2)}' > ${name}_host_mapped_2.read_ids.txt

Maybe in this case it doesn't necessarily get cleaner if solved like this?

skrakau · 2020-06-03T20:34:29Z

OK, I now added a test using the --host_fasta option and some brief documentation.

Currently this host removal only works for short reads. If --host_fasta or --host_genome is specified in combination with --manifest, an error with a corresponding message is returned. Is that OK?

skrakau · 2020-06-05T11:11:01Z

In the MAG test.config file temp_dir is set to a folder within outdir. So far I didn't do that for test_host_rm.config. Is this necessary for this pipeline, and if yes why?

skrakau · 2020-06-08T12:39:56Z

Hi @ewels or @apeltzer, could one of you tell me by any chance what the purpose of the line temp_dir = "./tests/tmp_dir" in the test.config files is (it's also used in the ampliseq pipline, but didn't find any documentation)? Still wondering if it is ok to just omit it in the new conf/test_host_rm.config file for this PR.

ewels · 2020-06-08T16:06:41Z

Link to line in question:

mag/conf/test.config

Line 18 in 4c2f61c

params.temp_dir = "./tests/tmp_dir"

Nothing to do with me I'm afraid. Looks like it was added by @HadrienG

Phil

ewels · 2020-06-08T16:07:01Z

(but I agree, I can't see anything obvious that it is doing, and I suspect that it can be removed from both config files)

skrakau · 2020-06-08T16:12:20Z

OK, thanks @ewels !

d4straub · 2020-06-09T07:52:44Z

Currently this host removal only works for short reads. If --host_fasta or --host_genome is specified in combination with --manifest, an error with a corresponding message is returned. Is that OK?

I think that's fine. Using proper settings for ONT qc (e.g. --longreads_length_weight 1) will remove those that have no matches with Illumina reads, so ONT reads can be filtered that way as well. And SPAdes anyway is using ONT reads only for resolving uncertainties in the assembly graph including gap closure and therefore ONT host reads are relatively unlikely to mess up the assembly, I think.

skrakau · 2020-06-09T17:24:56Z

Ok, will change it then so that the already filtered short reads will be used to filter the long reads.

…ost removal is run in combination with long reads.

skrakau · 2020-06-11T10:56:38Z

Hi @d4straub,

thanks for your input. The following points were added/changed now:

filtlong uses now the already filtered short reads as a reference (to indirectly filter out long host reads), for this I changed the channel files_all_raw to files_long_rawand joined this with the filtered short reads
accordingly I also changed the order of the processes in the code just for the sake of clarity: first short reads preprocessing, then long reads preprocessing
if host removal is turned on in combination with long reads, warnings are given out to inform the user that long reads are only filtered indirectly, and informing about the effect of --longreads_length_weight (if > 1)
added some information about this to the docs

I tested this locally for a very basic example, and with --longreads_length_weight 1 long reads mapping to host sequences were discarded as expected, reads mapping to the 'minigut' sequences not.

added missing parameters to the summary

Best,
Sabrina

d4straub · 2020-06-15T10:15:32Z

If I am not mistaken there are 3 tests now:

Illumina data
Illumina data + host removal
Illumina + ONT data

wouldn't it be good to test also

Illumina + ONT data + host removal

My reasoning is that there are now some channels that are not tested otherwise. And further changes breaking processes that use these channels might go undetected when the test is missing. What do you think?

edit: layout

…read removal

skrakau · 2020-06-16T11:00:08Z

Ok, I added a corresponding test, but it is more for channel testing, as I didn't add host reads to the long read dataset.

d4straub

Great, thanks! Looks good!

…ucible results

skrakau · 2020-06-18T11:17:01Z

I made the saving of the host read ids optional and added some sorting, since the order of the Bowtie 2 results is not reproducible.

d4straub

LGTM

Requested changes were either addressed or reasonably dismissed

skrakau added 5 commits May 30, 2020 20:58

Added host removal with Bowtie 2

4ad79d1

Created two separate MultiQC fastqc sections, raw and after preproces…

0e32c17

…sing.

Added MultiQC section for host read removal with Bowtie 2 based on cu…

2192f58

…stom content file

Updated CHANGELOG

f1848b3

Fixed MultiQC handling for setting without host removal

b5b4ca3

skrakau requested review from HadrienG and d4straub May 31, 2020 12:00

ewels reviewed Jun 2, 2020

View reviewed changes

skrakau added 2 commits June 2, 2020 12:21

Publish also bowtie2 log files of the remove_host process.

3738e84

Removed generation of unnecessary log files within remove_host process.

c942c4e

HadrienG previously requested changes Jun 3, 2020

View reviewed changes

skrakau added 2 commits June 3, 2020 17:38

Added tests for host read removal

ea775e4

Fixed handling of file

86ee5c2

skrakau force-pushed the add_host_removal branch from 071ae48 to 86ee5c2 Compare June 3, 2020 15:40

Added some documentation for host read removal

225aba6

skrakau requested a review from HadrienG June 4, 2020 12:46

skrakau added 2 commits June 10, 2020 20:58

Run filtlong together with already filtered short reads and warn if h…

d48a601

…ost removal is run in combination with long reads.

Added missing parameters to summary

e3db180

skrakau force-pushed the add_host_removal branch from 560b7c3 to e3db180 Compare June 10, 2020 19:02

Merge branch 'dev' into add_host_removal

ec0666e

Adjusted CHANGELOG

e2e63e2

d4straub reviewed Jun 15, 2020

View reviewed changes

Comment thread conf/test_host_rm.config

Added test for combination of hybrid (short and long reads) and host …

e879269

…read removal

d4straub approved these changes Jun 16, 2020

View reviewed changes

Made saving of host read ids optional and added sort to ensure reprod…

4c67f45

…ucible results

d4straub approved these changes Jun 18, 2020

View reviewed changes

skrakau merged commit 22464a2 into nf-core:dev Jun 18, 2020

skrakau deleted the add_host_removal branch May 31, 2021 13:36

		zcat ${reads[0]} \| echo "Read pairs before removal: \$((`wc -l`/4))" >>${name}_remove_host.log
		zcat ${name}_host_unmapped_1.fastq.gz \| echo "Read pairs after removal: \$((`wc -l`/4))" >>${name}_remove_host.log

Conversation

skrakau commented May 31, 2020

PR checklist

Uh oh!

d4straub commented Jun 2, 2020

Uh oh!

skrakau commented Jun 2, 2020

Uh oh!

ewels Jun 2, 2020

Choose a reason for hiding this comment

Uh oh!

skrakau Jun 2, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

HadrienG Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

skrakau Jun 3, 2020

Choose a reason for hiding this comment

Uh oh!

skrakau commented Jun 3, 2020

Uh oh!

skrakau commented Jun 5, 2020

Uh oh!

skrakau commented Jun 8, 2020

Uh oh!

ewels commented Jun 8, 2020

Uh oh!

ewels commented Jun 8, 2020

Uh oh!

skrakau commented Jun 8, 2020

Uh oh!

d4straub commented Jun 9, 2020

Uh oh!

skrakau commented Jun 9, 2020

Uh oh!

skrakau commented Jun 11, 2020

Uh oh!

Uh oh!

d4straub commented Jun 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

skrakau commented Jun 16, 2020

Uh oh!

d4straub left a comment

Choose a reason for hiding this comment

Uh oh!

skrakau commented Jun 18, 2020

Uh oh!

d4straub left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

d4straub commented Jun 15, 2020 •

edited

Loading