[This issue is to collect data regarding improvements to the sketching methodology for DNA (not AA/3Di)].

Datasets

Focusing on reads (most complicated use case) and using a metagenomic and an isolate example.

• Isolate: reference ERR9281752.
• Metagenomic: reference SRR5918785.

Code used

• Vanilla: sketchlib.rust master branch (commit b9a9310) of this repo, with a small change over it so that min_qual==0 avoids making the checks completely (in src/hashing/nthash_iterator.rs).
• SIMD: sketchlib.rust simdsketch_crate branch (commit 87192a5) of my fork. It is an implementation of sketchlib.rust, with the following caveats:
  • Not optimised, improvements to be done yet. Most surely not best way of doing it.
  • Bin size changed from 14 to 16, to accommodate simd_sketch.
  • No counts apart from the 0/1 argument (duplicate) possible.
  • Coverage set manually to 30 when the input file is a read.
  • Densification not applied.
• SIMD with final code: sketchlib.rust simdsketch_crate branch (commit fa8e72d) of my fork. It is the code from the previous point, with a cargo update done the 04/12/2025, and with some improvements:
  • Now we can set a minimum number of counts.
  • Estimated coverage is now an argument. We are using 15000 as the default for the numbers shown below.

Comparisons

• Default (equivalent to --min-count 5 --min-qual 20).
• Remove all count checks (--min-count 0).
• --min-count 2.
• Remove all quality checks (--min-qual 0).
• Remove all count and quality checks (--min-count 0, --min-qual 0).

Other arguments

• k = 21.
• sketch_size = 1024.
• threads = 1.

Commands

# Default 
sketchlib sketch -k $k -o test -f $tsvreadfile --threads $nthreads --verbose

# w/o counts
sketchlib sketch -k $k --min-count 0 -o test -f $tsvreadfile --threads $nthreads --verbose

# w/ counts = 2
sketchlib sketch -k $k --min-count 2 -o test -f $tsvreadfile --threads $nthreads --verbose

# w/o quality 
sketchlib sketch -k $k --min-qual 0 -o test -f $tsvreadfile --threads $nthreads --verbose

# w/o counts w/o quality 
sketchlib sketch -k $k --min-qual 0 --min-count 0 -o test -f $tsvreadfile --threads $nthreads --verbose

For the last version, the argument --est-coverage $ecov is added, with ecov=15000.

Hardware

• Intel Xeon Gold 6230 CPU @ 2.10GHz (2x, 80 threads).
• 800GB RAM.

Results

Time is recorded as that reported from the whole sketchlib run (sketchlib done in Xs), and is shown in the last two columns, in seconds.