Repository navigation
Description
Hi everybody. First of all, gemini is a great tool.
I am having a problem loading a full genome vcf from a trio (father, mother and son).
I am using gemini devel version 0.30
the comand is this
gemini load -v filtered_normalized_annotated.vcf -p P_Trio.ped -t snpEff --cores 3 Trio_1.db
Everything goes smoothly until this error arises
pid 8503: 239999 variants processed.
pid 8506: 239999 variants processed.
pid 8509: 249999 variants processed.
Traceback (most recent call last):
File "/usr/local/bin/gemini", line 7, in
gemini_main.main()
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
args.func(parser, args)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 311, in loadchunk_fn
gemini_load_chunk.load(parser, args)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 918, in load
gemini_loader.populate_from_vcf()
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 223, in populate_from_vcf
(variant, variant_impacts) = self._prepare_variation(var, anno_keys)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 407, in _prepare_variation
clinvar_info = annotations.get_clinvar_info(var)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/annotations.py", line 648, in get_clinvar_info
clinvar.clinvar_sig = info_map['CLNSIG'].lower()
KeyError: 'CLNSIG'
pid 8506: 249999 variants processed.
pid 8509: 259999 variants processed.
pid 8506: 259999 variants processed.
It looks like that the chunk that is processed in pid 8503, fails completely and is not resumed in the process.
I have the lastest versions of the annotations databases from gemini (update --dataonly). I have used other vcf from other trios (exome instead of genome), and this problem did not appear.
Could it be a problem within the vcf (a problematic snp?) or a bug in the devel version of gemini?
I am completely lost.
As i said before, i have used the same thing with other trios vcfs (from exome) without any problem.
Activity
CarlosGAH commentedon Feb 5, 2019
I forgot to mention, that error do not interrupt the loading, but at the end the proccess fails
ValueError: Processing failed on GEMINI chunk load
brentp commentedon Feb 5, 2019
thanks for reporting. can you report the output of
ls clin*in your gemini data directory that contains all the vcfs and bed annotation files?CarlosGAH commentedon Feb 5, 2019
Here it is
/usr/local/share/gemini/gemini_data$ ls clin*
clinvar_20170130.tidy.vcf.gz clinvar_20170130.tidy.vcf.gz.tbi clinvar_20190102.tidy.vcf.gz clinvar_20190102.tidy.vcf.gz.tbi
brentp commentedon Feb 5, 2019
would you try manually removing
clinvar_20170130.tidy.vcf.gzandclinvar_20170130.tidy.vcf.gz.tbito make sure gemini is not getting an older version?CarlosGAH commentedon Feb 5, 2019
ok, i will try and i will inform again
CarlosGAH commentedon Feb 5, 2019
The same problem appears
I have removed the archives as you said. But the problem persists
This problem do not arise in smaller vcfs (for example trios from exome sequencing). But in this big vcfs (trios from whole genome), the problem appears. If i select a fraction of this big vcf (1/10 for example), the problem do not appear
viktorstr commentedon Feb 5, 2019
Dear Brent,
i have the same problem - many CLNSIG errors during vcf loading:
Traceback (most recent call last):
File "/home/viktor/gemini/bin/gemini", line 7, in
gemini_main.main()
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
args.func(parser, args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 311, in loadchunk_fn
gemini_load_chunk.load(parser, args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 918, in load
gemini_loader.populate_from_vcf()
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 223, in populate_from_vcf
(variant, variant_impacts) = self._prepare_variation(var, anno_keys)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 407, in _prepare_variation
clinvar_info = annotations.get_clinvar_info(var)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/annotations.py", line 648, in get_clinvar_info
clinvar.clinvar_sig = info_map['CLNSIG'].lower()
KeyError: 'CLNSIG'
and finally it crash at the end:
Traceback (most recent call last):
File "/home/viktor/gemini/bin/gemini", line 7, in
gemini_main.main()
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
args.func(parser, args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 204, in load_fn
gemini_load.load(parser, args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 49, in load
load_multicore(args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 93, in load_multicore
chunks = load_chunks_multicore(grabix_file, args)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 264, in load_chunks_multicore
wait_until_finished(procs)
File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 359, in wait_until_finished
raise ValueError("Processing failed on GEMINI chunk load")
ValueError: Processing failed on GEMINI chunk load
Any ideas ?
Thanks
CarlosGAH commentedon Feb 5, 2019
I have tried the same big vcf that gave me probems with gemini 20.1 (clean installation from gemini_install.py in a new computer, updated dataonly and cadd y gerp scores), and it gave no problems at all. The problem is that the databases are a bit old. There is a way to update only the databases (for example clinvar and dbSNP?)
brentp commentedon Feb 5, 2019
can you get me the portion of the vcf that will recreate the error? I know you said on a small subset you do not see it, but you should be able to find 1 chunk of the file that gives the error. then I can debug and fix this problem for anyone who might encounter it.
viktorstr commentedon Feb 5, 2019
There is link for test vcf download. Hope it helps.
http://www.uschovna.cz/en/zasilka/JT42PY9VDAMSXGTN-SGU/?set_lang=en
Best regards
Viktor
CarlosGAH commentedon Feb 6, 2019
Here is the portion of the vcf that gave the problem/error
gemini_problem.vcf.gz
brentp commentedon Feb 7, 2019
thank you very much for the test-case. I have a push for this that will be pushed shortly.
fix for #917
brentp commentedon Feb 7, 2019
this is now fixed in master and will be a part of the release.