A fast LZ77-style block / streaming compressor — Rust port of github.com/minio/minlz.

This implements the MinLZ specification v1.0 (block + stream codec, including the optional stream index for random access). Block search tables (SPEC.md §4.13) are not implemented.

• Three compression levels — Level::Fastest (1), Level::Balanced (2, default), Level::Smallest (3).
• Block codec — encode / decode / try_encode / append_encoded / append_decoded / decoded_len / is_minlz / max_encoded_len. Blocks are capped at 8 MiB per the spec.
• Stream codec — stream::Reader<R: Read> / stream::Writer<W: Write>, multi-block framing with CRC32C integrity per block.
• Multi-threaded streaming — stream::MtWriter<W> parallelises encoding across worker threads; Reader::decode_concurrent parallelises decoding.
• Random access — index::Index plus stream::ReadSeeker give you uncompressed-offset seeks against any Read + Seek source. The index can be appended to the stream, stored as a sidecar, or recovered post-hoc with index::index_stream over a sequential reader.
• User-defined chunks — embed metadata inline in a stream (chunk IDs 0x80–0xFD, skippable and non-skippable).
• CRC32C — hardware-accelerated on x86_64 (SSE4.2) and aarch64 (CRC32 extension), with a portable table fallback.
• Safe public API — unsafe is confined to the hot path of the block decoder and a few load/store helpers; the crate itself is #![deny(missing_docs)] and CI runs Clippy + Miri + cargo-fuzz.

Add the dependency:

[dependencies]
minlz = { git = "https://github.com/minio/minlz-rs" }

Blocks are best for small payloads (≤ 8 MiB). They carry no integrity check — wrap them in a stream if corruption matters.

use minlz::{encode, decode, Level};

fn roundtrip(src: &[u8]) -> Result<(), minlz::Error> {
    let mut compressed = Vec::new();
    encode(&mut compressed, src, Level::Balanced)?;

    let mut decoded = Vec::new();
    decode(&mut decoded, &compressed)?;
    assert_eq!(decoded, src);
    Ok(())
}

Streams are independent blocks with framing, CRC32C, an EOF marker, and (optionally) an appended index. Both Writer and Reader are std::io::Write / std::io::Read, so they plug into anything in the ecosystem.

use std::io::{Read, Write};
use minlz::stream::{Reader, Writer};

fn roundtrip(src: &[u8]) -> std::io::Result<Vec<u8>> {
    let mut compressed: Vec<u8> = Vec::new();
    let mut w = Writer::new(&mut compressed);
    w.write_all(src)?;
    w.finish()?;                              // flush + EOF

    let mut out = Vec::new();
    Reader::new(&compressed[..]).read_to_end(&mut out)?;
    Ok(out)
}

WriterBuilder exposes .level(), .block_size(), .padding(), .append_index(), .uncompressed(), etc. ReaderBuilder lets you cap max_block_size, opt out of CRC checks for speed, and register callbacks for user chunks.

MtWriter and Reader::decode_concurrent saturate all cores with a single worker pool. MtWriter requires W: Send + 'static because it moves the sink into the writer thread and returns it via finish.

use std::io::Write;
use minlz::stream::{MtWriter, Reader};

fn parallel_roundtrip(payload: &[u8]) -> std::io::Result<()> {
    let mut w = MtWriter::new(Vec::<u8>::new());
    w.write_all(payload)?;
    let compressed = w.finish()?;

    let threads = std::thread::available_parallelism()?.get();
    let mut reader = Reader::new(&compressed[..]);
    let (n, _sink) = reader.decode_concurrent(Vec::<u8>::new(), threads)?;
    println!("decoded {n} bytes");
    Ok(())
}

use std::io::{Cursor, Read, Seek, SeekFrom, Write};
use minlz::stream::{ReadSeeker, Reader, WriterBuilder};

fn random_read(payload: &[u8], offset: u64) -> std::io::Result<Vec<u8>> {
    let mut compressed: Vec<u8> = Vec::new();
    let mut w = WriterBuilder::new()
        .append_index()                  // index chunk after EOF
        .build(&mut compressed);
    w.write_all(payload)?;
    w.finish()?;

    // Empty slice → ReadSeeker loads the index from the tail itself.
    let reader = Reader::new(Cursor::new(compressed));
    let mut rs = ReadSeeker::new(reader, &[])?;
    rs.seek(SeekFrom::Start(offset))?;
    let mut buf = vec![0u8; 4096];
    rs.read_exact(&mut buf)?;
    Ok(buf)
}

Working examples for these patterns live under crates/minlz/examples/ — see index_random_access.rs and index_sidecar.rs for the full plumbing.

A complementary knob is WriterBuilder::block_size (8 KiB–8 MiB, default 2 MiB). Smaller blocks trade ratio for lower memory and slightly higher per-block overhead; larger blocks give the L2/L3 encoders more matching range.

CockroachDB log, GitHub events JSON, GitHub ranks binary, NYC taxi CSV, VM Image.

The MT rows use stream::MtWriter for encode and Reader::decode_concurrent for decode at 32 worker threads. Encode scales 7–9× over the single-threaded baseline; decode scales 7–8× and sustains ~27–29 GB/s into an io::sink() target.

Headlines: at Fastest, MinLZ matches Snappy/LZ4 encode speed with substantially better ratio (2.78× → 2.17× on the Snappy set; 4.45× → 3.31× on real workloads) and roughly doubles their decode speed against Snappy. LZ4 wins raw decode throughput on large workloads; everything else trails MinLZ. Balanced lands near zstd-1 on speed but with mid-range LZ ratio. Smallest trades encoder throughput for ratios near zstd-1 while keeping decode close to MinLZ-1.

The mz binary at bin/mz/ is a Rust port of the Go cmd/mz tool.

Build or install from a checkout of this repo:

cargo install --path bin/mz            # installs to ~/.cargo/bin/mz
cargo build --release -p mz            # or just build; binary at target/release/mz

mz c -2 input.json                     # compress to input.json.mz
mz d input.json.mz                     # decompress back to input.json
mz cat input.json.mz                   # decompress to stdout
mz bench -3 input.json                 # encode + decode N=5 times, print MB/s
mz d --offset=1G+nl input.json.mz      # seek to 1 GiB, advance to newline
mz verify input.json.mz                # validate without writing output

mz --help lists every flag.

This crate implements the MinLZ specification v1.0. Subset features not implemented (versus the Go reference):

• Snappy / S2 fallback decoding for streams whose first byte indicates a legacy magic.
• Block search tables (SPEC.md §4.13).

cargo test --workspace               # 200+ unit + integration tests
cargo doc --no-deps                  # rustdoc
cargo bench -p minlz --bench block   # criterion: block codec
cargo bench -p minlz --bench stream  # criterion: stream codec (incl. MT and seek)

Fuzzing lives at crates/minlz/fuzz/ with five targets (decode-arbitrary, roundtrip, stream-roundtrip, stream-decode-arbitrary, index-load). See its RUNBOOK.md for setup.

MSRV: Rust 1.85.

Apache License 2.0 — same as the upstream Go implementation.