Skip to content

Support for checkpointing or resumable vg autoindex or minimizer for large Giraffe indexes #4871

@eabebe-tech

Description

@eabebe-tech

Hello vg team,

I'm building Giraffe indexes on an HPC cluster with a hard 48 hr walltime limit per job. The job used ~800GB RAM, it does not produce any output files until completion and exceeds the 48 hour walltime and gets killed, I am also using -t/threads to make it run faster.

Questions

  1. I wanted to ask if vg minimizer supports any form of checkpointing or resumable execution?
  2. Is there a way to make it periodically write partial results to disk?
  3. would splitting the graph by chromosome and merging minimizer indexes later be valid?

Enviorment:
Threads: 32
Memory: 800GB
running on apptainer in slurm

Thank you so much!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions