Skip to content
paddorPublic

About

Fast, memory-safe Rust zstd codec with all-level decode, -7..4 encode, and dictionary training.

Topics

Resources

Security policy

Stars

114 stars

Watchers

0 watching

Forks

Repository files navigation

zrip

Pure Rust zstd codec. The decoder reads standard zstd blocks and frames produced at any zstd compression level. The encoder supports levels -7 through 4 for fast transfer pipelines. First-class dictionary support.

Why zrip

  • Speed. Negative levels beat C zstd's speed at the same ratio on most files. See Performance.
  • No unsafe. Every crate is #![forbid(unsafe_code)]. See SAFETY.md.
  • Small. About 15k lines of Rust. No levels above 4; their extra ratio matters for storage, not transfers.
  • Dictionaries. Built-in FastCOVER training.
  • Push API. Sans-io PushEncoder and PushDecoder for async codecs and protocol stacks. See Push.
  • Adaptive level. Picks the level at run time from what limits the transfer: output, CPU, or input. See Adaptive level.
  • no_std + alloc. The frame feature needs std.
  • WebAssembly. @paddor/zrip on JSR and npm, with WASM SIMD detection.

Performance

zstd pipeline benchmark

x86_64 details (per-file pipeline, scatter, matrix, small inputs)

per-file pipeline encode speed vs compression ratio per-file encode/decode matrix tiny input pipeline by compressibility small input encode throughput small input decode throughput

x86_64 dictionaries (Silesia slices from 64 B to 1 MiB, with and without a dictionary)

dictionary sweep at level 1 dictionary sweep at level 3

No current aarch64 charts. If you have an aarch64 machine, run the benchmark and chart commands from DEVELOPMENT.md and send a PR with doc/charts/aarch64/.

wasm32 (wasmtime)

wasm32 pipeline summary wasm32 per-file pipeline wasm32 encode speed vs compression ratio wasm32 per-file encode/decode matrix

API

Levels go from -7 (fastest) to 4. Level 0 means the default, level 1. DESIGN.md lists each level's parameters.

// One-shot (allocating)
let compressed = zrip::compress(input, 1)?;
let original   = zrip::decompress(&compressed)?;

// One-shot into caller buffer
let n = zrip::compress_into(input, &mut output_buf, 1)?;
zrip::decompress_into(&compressed, &mut output_vec)?;

// Reusable context (amortizes table allocation across calls)
let mut ctx = zrip::CompressContext::new(1)?;
let compressed = ctx.compress(input)?;

let mut dec = zrip::DecompressContext::new();
let original = dec.decompress(&compressed)?;

// Reuse context state while writing into caller-owned buffers.
let mut frame = vec![0u8; zrip::compress_bound(input.len())];
let n = ctx.compress_into(input, &mut frame)?;
let mut owned = Vec::new();
dec.decompress_into(&frame[..n], &mut owned)?;

Streaming

use std::io::Write;

let mut enc = zrip::FrameEncoder::new(Vec::new(), 1)?;
enc.write_all(b"hello")?;
enc.write_all(b" world")?;
let compressed = enc.finish()?;

use std::io::Read;
let mut dec = zrip::FrameDecoder::new(&compressed[..]);
let mut out = String::new();
dec.read_to_string(&mut out)?;

FrameEncoder and FrameDecoder own persistent hash tables and workspace buffers. Call reset() to start a new frame while reusing all allocations:

let mut enc = zrip::FrameEncoder::new(Vec::new(), 1)?;
enc.write_all(b"first frame")?;
let first  = enc.reset(Vec::new())?;  // finishes frame, keeps buffers
enc.write_all(b"second frame")?;
let second = enc.finish()?;

Push (sans-io)

PushEncoder and PushDecoder take input in chunks of any size and write into caller buffers of any size, like zstd's ZSTD_compressStream2 and ZSTD_decompressStream. Use them where Read/Write don't fit: async codecs, HTTP bodies, sans-io protocol stacks. They need only alloc.

use zrip::push::{EndOp, PushDecoder, PushEncoder, Status};

let mut enc = PushEncoder::new(1)?;
let mut frame = Vec::new();
for chunk in chunks {
    enc.encode_to_vec(chunk, &mut frame, EndOp::Continue)?;
}
enc.encode_to_vec(&[], &mut frame, EndOp::End)?;

let mut dec = PushDecoder::new();
let mut buf = [0u8; 4096];
let p = dec.decode(&frame, &mut buf)?;
// p.consumed, p.produced, p.status: NeedsInput, NeedsOutput or Finished

decode stops at each frame end with Status::Finished; call again for the next frame. At end of input, finish() rejects a truncated frame. PushDecoder::max_window() sets the window cap (128 MiB by default), and with it the history kept.

Large-window and long distance matching

The normal match finder operates within a sliding window (512 KiB at L1). LDM finds matches at distances up to 1 << window_log bytes by sampling positions into a separate hash table. Useful for data with long-range repeats: log files, database dumps, source archives. See DESIGN.md for how LDM works.

let opts = zrip::Options::default().window_log(24).ldm(true);
let compressed = zrip::compress_opts(input, 1, &opts)?;

Streaming with LDM:

use std::io::Write;
let opts = zrip::Options::default().window_log(24).ldm(true);
let mut enc = zrip::FrameEncoder::with_options(Vec::new(), 1, &opts)?;
enc.write_all(input)?;
let compressed = enc.finish()?;

.ldm(true) without window_log uses a 128 MiB window (window_log 27), like zstd --long. One-shot calls shrink it to the input. FrameEncoder does not know its input size, so its frames always advertise 128 MiB, even short ones, and decoders must allow that window. Set window_log for a smaller window.

window_log controls the maximum match distance (24 = 16 MiB, 27 = 128 MiB). One-shot frames up to 128 MiB advertise their content size as the window, so there LDM can reach the whole input. The main cost is memory: the encoder allocates an LDM hash table of up to 8 MiB. FrameEncoder also buffers up to twice the window, and FrameDecoder keeps up to two windows. On a long stream at window_log=27 that is ~275 MiB to encode and ~260 MiB to decode. On data without long-range repeats, LDM adds overhead with no ratio benefit.

Dictionary compression

let dict = zrip::Dictionary::from_bytes(&dict_bytes)?;
let compressed = zrip::compress_with_dict(input, 1, &dict)?;
let original   = zrip::decompress_with_dict(&compressed, &dict)?;

Streaming with a dictionary:

let mut enc = zrip::FrameEncoder::with_dict(Vec::new(), 1, dict.clone())?;
enc.write_all(input)?;
let compressed = enc.finish()?;

let mut dec = zrip::FrameDecoder::with_dict(&compressed[..], dict);
let mut output = Vec::new();
dec.read_to_end(&mut output)?;

CompressContext::with_dict() and DecompressContext::with_dict() provide the same reuse for one-shot compression.

Dictionary training

Build a dictionary from sample data using the built-in FastCOVER trainer. Requires the dict_builder feature.

use zrip::dict::{train_dict_fastcover, fastcover::FastCoverParams};

let samples: Vec<&[u8]> = messages.iter().map(|m| m.as_bytes()).collect();
let dict = train_dict_fastcover(&samples, 16384, FastCoverParams::default());
// Use with compress_with_dict() / decompress_with_dict()

Adaptive level

zrip::adaptive::AdaptiveLevel picks the level at run time. The caller reports each window's compression work and what limited it: output, CPU, or idle input. It settles on the best measured level, which the caller applies with CompressContext::set_level().

Features

Feature Default Description
std yes Enables FrameEncoder, FrameDecoder
frame yes Frame header parsing and writing; implies std
alloc yes no_std + heap via alloc crate; includes CompressContext, DecompressContext
ldm yes Long distance matching for large-window compression
dict_builder no FastCOVER dictionary training
simd yes fearless_simd runtime dispatch (AVX2+BMI2, NEON, SIMD128)
paranoid no Keeps any future unsafe out; no effect today
nightly no #[optimize] attributes on hot functions

Safety

SAFETY.md describes how zrip stays fast without unsafe and catalogs C zstd memory safety bugs that Rust prevents by construction.

All codec paths are fuzz-tested (21 targets). Fuzz targets cover round-trip correctness, cross-validation against C zstd, streaming, dictionary modes, push chunking, and corruption resistance (bitflip, splice, truncate, overwrite). A differential test and fuzz target check that the decoder accepts exactly the frames C zstd accepts; DEVELOPMENT.md lists the one exception.

Design

DESIGN.md covers the encode/decode pipeline, performance architecture, SIMD dispatch, compile-time specialization, and divergences from C zstd.

License

MIT. See LICENSE.

About

Fast, memory-safe Rust zstd codec with all-level decode, -7..4 encode, and dictionary training.

Topics

Resources

Security policy

Stars

114 stars

Watchers

0 watching

Forks

Releases

Contributors

Languages