Skip to content

Problem during gemini load #917

Closed
Closed

Description

@CarlosGAH

Hi everybody. First of all, gemini is a great tool.
I am having a problem loading a full genome vcf from a trio (father, mother and son).
I am using gemini devel version 0.30
the comand is this
gemini load -v filtered_normalized_annotated.vcf -p P_Trio.ped -t snpEff --cores 3 Trio_1.db

Everything goes smoothly until this error arises
pid 8503: 239999 variants processed.
pid 8506: 239999 variants processed.
pid 8509: 249999 variants processed.
Traceback (most recent call last):
File "/usr/local/bin/gemini", line 7, in
gemini_main.main()
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
args.func(parser, args)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 311, in loadchunk_fn
gemini_load_chunk.load(parser, args)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 918, in load
gemini_loader.populate_from_vcf()
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 223, in populate_from_vcf
(variant, variant_impacts) = self._prepare_variation(var, anno_keys)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 407, in _prepare_variation
clinvar_info = annotations.get_clinvar_info(var)
File "/usr/local/share/gemini/anaconda/lib/python2.7/site-packages/gemini/annotations.py", line 648, in get_clinvar_info
clinvar.clinvar_sig = info_map['CLNSIG'].lower()
KeyError: 'CLNSIG'
pid 8506: 249999 variants processed.
pid 8509: 259999 variants processed.
pid 8506: 259999 variants processed.

It looks like that the chunk that is processed in pid 8503, fails completely and is not resumed in the process.
I have the lastest versions of the annotations databases from gemini (update --dataonly). I have used other vcf from other trios (exome instead of genome), and this problem did not appear.
Could it be a problem within the vcf (a problematic snp?) or a bug in the devel version of gemini?
I am completely lost.
As i said before, i have used the same thing with other trios vcfs (from exome) without any problem.

Activity

  1. CarlosGAH commented on Feb 5, 2019

    @CarlosGAH
    Author

    I forgot to mention, that error do not interrupt the loading, but at the end the proccess fails
    ValueError: Processing failed on GEMINI chunk load

  2. brentp commented on Feb 5, 2019

    @brentp
    Collaborator

    thanks for reporting. can you report the output of ls clin* in your gemini data directory that contains all the vcfs and bed annotation files?

  3. CarlosGAH commented on Feb 5, 2019

    @CarlosGAH
    Author

    Here it is
    /usr/local/share/gemini/gemini_data$ ls clin*
    clinvar_20170130.tidy.vcf.gz clinvar_20170130.tidy.vcf.gz.tbi clinvar_20190102.tidy.vcf.gz clinvar_20190102.tidy.vcf.gz.tbi

  4. brentp commented on Feb 5, 2019

    @brentp
    Collaborator

    would you try manually removing clinvar_20170130.tidy.vcf.gz and clinvar_20170130.tidy.vcf.gz.tbi to make sure gemini is not getting an older version?

  5. CarlosGAH commented on Feb 5, 2019

    @CarlosGAH
    Author

    ok, i will try and i will inform again

  6. CarlosGAH commented on Feb 5, 2019

    @CarlosGAH
    Author

    The same problem appears
    I have removed the archives as you said. But the problem persists
    This problem do not arise in smaller vcfs (for example trios from exome sequencing). But in this big vcfs (trios from whole genome), the problem appears. If i select a fraction of this big vcf (1/10 for example), the problem do not appear

  7. viktorstr commented on Feb 5, 2019

    @viktorstr

    Dear Brent,
    i have the same problem - many CLNSIG errors during vcf loading:
    Traceback (most recent call last):
    File "/home/viktor/gemini/bin/gemini", line 7, in
    gemini_main.main()
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
    args.func(parser, args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 311, in loadchunk_fn
    gemini_load_chunk.load(parser, args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 918, in load
    gemini_loader.populate_from_vcf()
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 223, in populate_from_vcf
    (variant, variant_impacts) = self._prepare_variation(var, anno_keys)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load_chunk.py", line 407, in _prepare_variation
    clinvar_info = annotations.get_clinvar_info(var)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/annotations.py", line 648, in get_clinvar_info
    clinvar.clinvar_sig = info_map['CLNSIG'].lower()
    KeyError: 'CLNSIG'

    and finally it crash at the end:

    Traceback (most recent call last):
    File "/home/viktor/gemini/bin/gemini", line 7, in
    gemini_main.main()
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 1249, in main
    args.func(parser, args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_main.py", line 204, in load_fn
    gemini_load.load(parser, args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 49, in load
    load_multicore(args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 93, in load_multicore
    chunks = load_chunks_multicore(grabix_file, args)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 264, in load_chunks_multicore
    wait_until_finished(procs)
    File "/home/viktor/gemini/anaconda/lib/python2.7/site-packages/gemini/gemini_load.py", line 359, in wait_until_finished
    raise ValueError("Processing failed on GEMINI chunk load")
    ValueError: Processing failed on GEMINI chunk load

    Any ideas ?
    Thanks

  8. CarlosGAH commented on Feb 5, 2019

    @CarlosGAH
    Author

    I have tried the same big vcf that gave me probems with gemini 20.1 (clean installation from gemini_install.py in a new computer, updated dataonly and cadd y gerp scores), and it gave no problems at all. The problem is that the databases are a bit old. There is a way to update only the databases (for example clinvar and dbSNP?)

  9. brentp commented on Feb 5, 2019

    @brentp
    Collaborator

    can you get me the portion of the vcf that will recreate the error? I know you said on a small subset you do not see it, but you should be able to find 1 chunk of the file that gives the error. then I can debug and fix this problem for anyone who might encounter it.

  10. viktorstr commented on Feb 5, 2019

    @viktorstr

    There is link for test vcf download. Hope it helps.

    http://www.uschovna.cz/en/zasilka/JT42PY9VDAMSXGTN-SGU/?set_lang=en
    Best regards
    Viktor

  11. CarlosGAH commented on Feb 6, 2019

    @CarlosGAH
    Author

    Here is the portion of the vcf that gave the problem/error
    gemini_problem.vcf.gz

  12. brentp commented on Feb 7, 2019

    @brentp
    Collaborator

    thank you very much for the test-case. I have a push for this that will be pushed shortly.

  13. added a commit that references this issue on Feb 7, 2019
  14. brentp commented on Feb 7, 2019

    @brentp
    Collaborator

    this is now fixed in master and will be a part of the release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions