Pure Rust implementation of archive and compression formats with core algorithms implemented from scratch.
OxiArc is a comprehensive archive/compression library and CLI tool written in pure Rust. It provides support for multiple archive formats and compression algorithms, all implemented without relying on C bindings or external compression libraries. Built from the ground up with performance and safety in mind.
- ZIP - PKZIP format with DEFLATE and Store methods, Zip64 support
- TAR - POSIX tar with UStar and PAX extended headers
- GZIP - GNU zip single-file compression (RFC 1952)
- LZH/LHA - Japanese archive format with lh0, lh1, lh4, lh5, lh6, lh7, lhd methods
- XZ - Modern LZMA2 compression format
- 7z - 7-Zip archive format (read-only)
- CAB - Microsoft Cabinet format (read-only)
- LZ4 - Fast LZ4 frame format
- Zstandard - Facebook's fast compression format
- Bzip2 - Block-sorting compression
- Brotli - Brotli compression (RFC 7932)
- Snappy - Google's fast compression format
- ISO 9660 - CD/DVD disc image format (read-only)
- DEFLATE (RFC 1951) - LZ77 + Huffman, levels 0-9, async deflate support
- LZMA/LZMA2 - Range coding with context modeling
- LZH - LZSS + Huffman (lh0, lh4, lh5, lh6, lh7) plus lh1/lh2/lh3 (LHarc 1.x/2.x adaptive and block-static Huffman), the LArc methods lzs/lz4/lz5, PMarc pm0, and lhd directory entries; pm1/pm2 are not implemented
- Bzip2 - BWT + MTF + RLE + Huffman
- LZ4 - Ultra-fast LZ77 variant with LZ4-HC
- Zstandard (RFC 8878) - Full decoder (FSE + 1/4-stream Huffman literals); encoder emits Huffman-compressed literals with predefined/RLE FSE sequence coding
- LZW - Lempel-Ziv-Welch for TIFF and GIF compression (MSB/LSB bitstream)
- Brotli (RFC 7932) - LZ77 + context-dependent Huffman, complete Appendix A static dictionary (122,784 bytes, all 121 transforms), quality 0-11
- Snappy - Ultra-fast LZ77 variant with block and framed formats
- Store - No compression
- AEC/SZIP (CCSDS-121.0-B-2) - Adaptive entropy coding for scientific datasets
- Pure Rust - No C/Fortran dependencies, 100% safe Rust
- Reference-Interop Verified - Every codec is validated by differential tests against the reference implementation, in both directions (see Reference-Implementation Differential Testing)
- Optimized CRC - Slicing-by-8 implementation (3-5x faster than table lookup)
- SIMD CRC32 - Hardware-accelerated CRC32 via aarch64 PMULL (Apple Silicon) and x86_64 PCLMULQDQ + SSE4.1
- Modern CLI - Progress bars, verbose output, JSON support, shell completions
- Streaming API - Memory-efficient processing with stdin/stdout support
- Async I/O - Async ZIP and async deflate support (async-io feature flag)
- Streaming API - GzipStream/ZlibStream/LzwStream encoders/decoders with flush modes
- Dry-Run Mode - Preview operations without writing files
- EntryBuilder - Fluent API for building archive entries
- Pattern Filtering - Include/exclude patterns with glob syntax
- Metadata Preservation - Timestamps, permissions, extended attributes
- Auto-detection - Automatic format detection from magic bytes
- Flexible Overwrite - Overwrite, skip, or prompt modes
- Progress/Cancel -
with_progressandwith_cancelbuilders on lz4, zstd, and lzma2 codecs - Optimal DEFLATE - Zopfli-style graph-based optimal parsing via
Deflater::with_optimal_parsing(level) - Bounded-Memory LZ4 - True streaming LZ4 with configurable memory budget via
with_memory_budget(usize) - Snappy Parallel - Rayon-based parallel frame compression via
parallelfeature - Memory-Mapped Files - Zero-copy
MappedFileprimitive in oxiarc-core (mmapfeature) - LZ4 Dict Blocks - Block-layer prefix dictionary support via
Lz4DictBlockEncoder/Lz4DictBlockDecoder,compress_block_with_dict,decompress_block_dict - Parallel GZIP - pigz-style multi-member parallel GZIP via
gzip_compress_parallel/ParallelGzipEncoder(parallelfeature in oxiarc-deflate) - LZ77 Tuning API - Fine-grained LZ77 heuristics via
Lz77ParamsandLz77Preset(nice_match + chain tuning) - DEFLATE Memory Pool - Thread-safe buffer reuse via
DeflatePool/PooledBuffor high-throughput workloads - Parallel LZMA2 - Multi-threaded LZMA2 compression via
lzma2_compress_parallel/ParallelLzma2Encoder(parallelfeature in oxiarc-lzma) - Raw-Preserve Append -
oxiarc addpreserves ZIP/LZH entries byte-for-byte (no re-compression) - ISO 9660 Read -
oxiarc list/extract/info/detectsupport for.isodisc images - Memory Limit -
--memory-limit <BYTES>option forextractandlist(e.g.--memory-limit 100M) - LZH/LZMA Dictionaries - Prefix dictionary support for LZH (
LzhEncoder::with_dictionary,LzhDecoder::with_dictionary) and LZMA (LzmaEncoder::with_dictionary,LzmaDecoder::with_dictionary) - LZMA Memory Pool - Thread-safe buffer reuse for LZMA decoders via
LzmaPool,PooledBuf,LzmaDecoderPooled(parallelfeature in oxiarc-lzma) - Archive Repair - Repair truncated/corrupted ZIP and TAR archives via
repair_zip,repair_tar,ZipRepair,TarRepair,RepairReport - Snappy Memory Pool - Thread-safe buffer reuse for Snappy FrameEncoder/FrameDecoder via
SnappyPool,PoolStats,compress_frame_pooled - Snappy Dictionaries - Block and frame level dictionary support (
compress_block_with_dict,compress_frame_with_dict,decompress_block_with_dict,decompress_frame_with_dict) - Snappy Async I/O - Async compression/decompression via
AsyncSnappyCompressor,AsyncSnappyDecompressor(async-iofeature in oxiarc-snappy) - Zstd Multi-Frame - Multi-frame decompression via
decompress_multi_frame,decompress_multi_frame_with_dict; streaming dict multi-frame fix - CLI Man Pages - Full set of troff
.1man pages for all CLI subcommands inman/directory - Snappy/Brotli Interop Tests - 35 new integration tests against wire-format golden vectors (16 Snappy, 19 Brotli) validating spec compliance
- AEC/SZIP Codec - CCSDS-121.0-B-2 compliant adaptive entropy coding via
oxiarc-szipwithBitReader/BitWriter,encode/decode/encode_bytesentry points,SzipParamsconfiguration,SzipErrorerror type - Non-Panicking Constructors -
RingBuffer::try_new/OutputRingBuffer::try_newfallible alternatives to the panicking constructors, for untrusted/arbitrary window sizes (oxiarc-core) - CLI Quiet Mode & Stdin Everywhere - Global
--quiet/-qflag;list/test/info/detectnow accept-for stdin (previously onlyextract/createdid) - Symlink-Aware Extraction - TAR entries that declare a symlink are recreated as real symlinks on extraction instead of being silently followed/overwritten
+----------------------------------------------------------+
| L4: Unified API (oxiarc-cli) |
| CLI with progress bars, verbose mode, filters |
+----------------------------------------------------------+
| L3: Container (oxiarc-archive) |
| ZIP, TAR, GZIP, LZH, XZ, 7z, CAB, LZ4, Zstd, Bzip2, Brotli, Snappy, ISO 9660 |
+----------------------------------------------------------+
| L2: Codecs |
| oxiarc-deflate: DEFLATE (RFC 1951) + async + GZip |
| oxiarc-lzma: LZMA/LZMA2 |
| oxiarc-lzhuf: LZH (lh0, lh1, lh4, lh5, lh6, lh7, lhd) |
| oxiarc-bzip2: BWT + MTF + Huffman |
| oxiarc-lz4: LZ4 block/frame |
| oxiarc-zstd: Zstandard (RFC 8878 FSE + Huffman) |
| oxiarc-lzw: LZW (GIF/TIFF, MSB/LSB bitstream) |
| oxiarc-brotli: Brotli (RFC 7932) |
| oxiarc-snappy: Snappy (block + framed) |
| oxiarc-szip: AEC/SZIP (CCSDS-121.0-B-2 adaptive entropy coding) |
+----------------------------------------------------------+
| L1: Core (oxiarc-core) |
| BitReader/Writer, RingBuffer, CRC-16/32/64 (simd-8) |
+----------------------------------------------------------+
| Crate | Description | Lines | Tests |
|---|---|---|---|
oxiarc-core |
Core primitives: BitStream (LSB + MSB), RingBuffer, CRC-16/32/64 (slicing-by-8), EntryBuilder, Serde | ~6,043 | 171 |
oxiarc-deflate |
DEFLATE (RFC 1951) + async deflate + GZip (multi-member) + true streaming (GzipStream/ZlibStream) | ~9,893 | 273 |
oxiarc-lzhuf |
LZH compression (lh0, lh1, lh4, lh5, lh6, lh7, lhd) with LZSS + Huffman + custom dictionaries | ~6,606 | 252 |
oxiarc-bzip2 |
Bzip2 with BWT + MTF + RLE + multi-table Huffman, multi-stream decode, de-randomisation | ~3,303 | 104 |
oxiarc-lz4 |
LZ4 block/frame + LZ4-HC with XXHash32, linked (block-dependent) frames, acceleration parameter | ~5,971 | 151 |
oxiarc-zstd |
Zstandard (RFC 8878) with FSE + Huffman + XXHash64, dictionary support, multi-frame | ~7,336 | 200 |
oxiarc-lzma |
LZMA/LZMA2 with range coding + hash chains + memory pool, multi-chunk .xz |
~7,957 | 178 |
oxiarc-archive |
13 container formats (ZIP, TAR, GZIP, LZH, XZ, 7z, CAB, LZ4, Zstd, Bzip2, Brotli, Snappy, ISO 9660) + async ZIP + archive repair | ~22,153 | 590 |
oxiarc-lzw |
LZW compression (GIF/TIFF incl. TIFF 6.0 Clear Code) with MSB/LSB bitstream, streaming encoder/decoder | ~2,775 | 93 |
oxiarc-brotli |
Brotli compression (RFC 7932) with the full Appendix A static dictionary, quality 0-11, streaming | ~7,153 | 237 |
oxiarc-snappy |
Snappy compression (block + framed format) with CRC32C, memory pool, dictionaries, async I/O | ~4,304 | 132 |
oxiarc-szip |
AEC/SZIP (CCSDS-121.0-B-2): encode/decode/encode_bytes, SzipParams, libaec-interoperable | ~1,902 | 46 |
oxiarc-cli |
CLI tool with progress bars, filters, JSON output, dry-run mode, enforced --memory-limit, man pages |
~6,897 | 92 |
| Total | Pure Rust archive/compression library | ~92,598 code lines (339 Rust files; 394 workspace-wide incl. fuzz/docs/scripts) | 2,519 |
Lines are tokei Rust code lines per crate (src + tests + examples); the Tests column is nextest (all-features, per-crate); the workspace additionally has 139 doctests (not attributed per crate), for 2,658 tests total. Measured 2026-08-06.
cargo install oxiarc-cligit clone https://github.com/cool-japan/oxiarc
cd oxiarc
cargo build --release
cargo install --path oxiarc-cli[dependencies]
oxiarc-archive = "0.4.1" # For archive format support
oxiarc-deflate = "0.4.1" # For DEFLATE compression
oxiarc-lzma = "0.4.1" # For LZMA/LZMA2 compression
oxiarc-bzip2 = "0.4.1" # For Bzip2 compression
oxiarc-lz4 = "0.4.1" # For LZ4 compression
oxiarc-zstd = "0.4.1" # For Zstandard compression
oxiarc-brotli = "0.4.1" # For Brotli compression
oxiarc-snappy = "0.4.1" # For Snappy compression
oxiarc-szip = "0.4.1" # For AEC/SZIP (CCSDS-121.0-B-2) compression# List archive contents
oxiarc list archive.zip
oxiarc list archive.7z --verbose
# Extract archives
oxiarc extract archive.zip
oxiarc extract data.tar.gz -o output/
oxiarc extract files.7z --progress
# Create archives
oxiarc create backup.zip file1.txt file2.txt folder/
oxiarc create data.tar dir1/ dir2/
oxiarc create compressed.xz large_file.bin
# Test integrity
oxiarc test archive.zip
oxiarc test data.lzh --verbose
# Show detailed information
oxiarc info archive.7z
oxiarc info data.cab
# Detect format
oxiarc detect unknown_file.bin
# Convert between formats
oxiarc convert old.lzh new.zip
oxiarc convert data.7z backup.taruse oxiarc_deflate::{deflate, inflate};
use oxiarc_archive::ZipReader;
use std::fs::File;
// Compress data with DEFLATE
let compressed = deflate(b"Hello, World!", 6)?;
let decompressed = inflate(&compressed)?;
// Read a ZIP archive
let file = File::open("archive.zip")?;
let mut zip = ZipReader::new(file)?;
for entry in zip.entries() {
println!("{}: {} bytes", entry.name, entry.size);
}The standard compression used in ZIP, GZIP, and PNG:
- LZ77 dictionary compression with 32KB sliding window
- Canonical Huffman coding
- Supports stored, fixed, and dynamic blocks
- Compression levels 0-9
Japanese archive format compression:
- LZSS with configurable window sizes (4KB-64KB)
- Static Huffman coding with dual trees (codes + offsets)
- Methods: lh0 (stored), lh1 (4KB window + adaptive Huffman), lh2 (8KB + adaptive Huffman), lh3 (8KB + block-static Huffman), lh4, lh5, lh6, lh7, lhd (directory), lzs/lz5 (LArc LZSS), lz4/pm0 (stored); pm1/pm2 are not implemented; unknown methods are listed and skipped per entry
- Shift_JIS filenames and level-2 headers (LHA 2.x standard) on write
Advanced compression used in 7z and XZ:
- LZ77-style dictionary compression
- Range coding for entropy encoding
- Context-dependent probability models
- 11-bit probability model (2048 states)
Block-sorting compression:
- Burrows-Wheeler Transform (BWT)
- Move-To-Front (MTF) coding
- Run-Length Encoding (RLE)
- Huffman coding
Ultra-fast compression:
- Simple LZ77 variant
- Block and frame formats
- Minimal CPU overhead
Modern fast compression (RFC 8878):
- Full decoder: FSE (predefined, RLE, custom
FSE_Compressed, and repeat modes) plus 1- and 4-stream Huffman literals — differentially verified byte-identical against referencezstd(incl. dictionary frames), with the RFC 8878 LIFO/MSB-first backward bitstream - Encoder: Huffman-compressed literal sections (chosen when they beat Raw/RLE, self-verified per section); sequences use the RFC 8878 predefined/RLE FSE tables — RFC-valid and accepted by
zstd -d, but custom block-optimal sequence tables are not emitted yet, so ratio on some inputs trails the reference encoder - XXHash64 checksums
- Dictionary support
Lempel-Ziv-Welch compression:
- GIF LZW codec with configurable initial code size
- LSB-first bitstream packing (GIF standard)
- MSB-first bitstream packing (TIFF standard) with TIFF 6.0 Clear Code semantics — interoperable with libtiff/Pillow/GDAL in both directions (oxiarc's encoded output is byte-identical to libtiff's)
- Variable bit widths (2-12 bits) with clear/EOI codes
Modern compression format:
- Full RFC 7932 decoder: block-type switching, context maps with the exact §7.1 context tables, the complete distance code space, metadata meta-blocks — differentially verified byte-identical against the reference
brotliCLI (qualities 0-11, windows 10-24) - The complete, byte-exact 122,784-byte Appendix A static dictionary with all 121 word transforms (UTF-8-aware ferment casing)
- RFC-conformant encoder accepted by
brotli -d; ratio trails the reference encoder at quality 10-11 and on structured binary data (no encode-side block-splitting/context modeling — a ratio limitation, not a correctness one) - Quality levels 0-11 (fast to best compression)
- Streaming compression/decompression API
Adaptive entropy coding for scientific data:
- CCSDS-121.0-B-2 standard implementation, differentially verified byte-identical against live libaec 1.1.4 in both directions
- Used in HDF5 and NetCDF scientific datasets
BitReader/BitWriterfor efficient bit manipulationSzipParamsstruct for encoding/decoding configurationencode/decode/encode_bytesentry points
| Crate | Status | Public API | Tests Passing |
|---|---|---|---|
| oxiarc-core | Stable | 228 | 171 |
| oxiarc-deflate | Stable | 168 | 273 |
| oxiarc-lzhuf | Stable | 106 | 252 |
| oxiarc-bzip2 | Stable | 56 | 104 |
| oxiarc-lz4 | Stable | 126 | 151 |
| oxiarc-zstd | Stable | 161 | 200 |
| oxiarc-lzma | Stable | 188 | 178 |
| oxiarc-archive | Stable | 438 | 590 |
| oxiarc-lzw | Stable | 67 | 93 |
| oxiarc-brotli | Stable | 101 | 237 |
| oxiarc-snappy | Stable | 35 | 132 |
| oxiarc-szip | Stable | 27 | 46 |
| oxiarc-cli | Stable | 45 | 92 |
| Total | 1,746 | 2,519 |
Test counts measured 2026-08-06 (nextest, all features, per-crate, 0 failed, 0 ignored; the workspace additionally has 139 doctests not attributed per crate, for 2,658 tests total); public-API item counts are the v0.3.6 snapshot. All crates are feature-complete and, as of the 2026-07-13 production-hardening campaign, validated against the reference implementation of every format in both directions. Ahead of a 1.0 release, 18 public format/method/status/error enums (FlushMode, CompressStatus/DecompressStatus, CompressionMethod, EntryType, ArchiveFormat, zstd BlockType/LiteralsBlockType, Lz4Level, the codec error enums, and more) are marked #[non_exhaustive] for forward-compatible matching.
Streaming compression/decompression support in oxiarc-deflate:
GzipStreamEncoder/GzipStreamDecoderwith configurable block sizesZlibStreamEncoder/ZlibStreamDecoderwith flush modes- Flush modes:
sync_flush,full_flush,partial_flush
| Format | Read | Write | Compression | Checksums | Notes |
|---|---|---|---|---|---|
| ZIP | ✅ | ✅ | DEFLATE, Store | CRC-32 | Zip64 support, data descriptors, async ZIP (async-io feature), AES-128/192/256 + ZipCrypto encryption (external encrypted archives detected via general-purpose bit 0); spanned/multi-volume ZIP unsupported (rejected) |
| TAR | ✅ | ✅ | N/A (container only) | None | UStar, PAX, GNU long names, GNU sparse read support for all three variants — old-format 'S', PAX 0.1, and PAX 1.0 — in both the seekable and streaming readers (writer-side sparse emission unsupported; sparse-source files are written as regular dense entries) |
| GZIP | ✅ | ✅ | DEFLATE | CRC-32 | RFC 1952 compliant |
| LZH | ✅ | ✅ | lh0-lh7, lzs, lz4, lz5, pm0 | CRC-16 | Shift_JIS support, all header levels; pm1/pm2 not implemented |
| XZ | ✅ | ✅ | LZMA2 | CRC-64 | Block checksums |
| 7z | ✅ | ❌ | LZMA/LZMA2 | CRC-32 | Read-only, partial support |
| CAB | ✅ | ❌ | None, MSZIP | CFDATA checksums | Microsoft Cabinet, read-only; MSZIP window carried across CFDATA blocks, per-block checksums validated; Quantum/LZX unsupported (clean error, never silent raw copy) |
| LZ4 | ✅ | ✅ | LZ4, LZ4-HC | XXHash32 | Frame format, block/content checksums |
| Zstd | ✅ | ✅ | Zstandard | XXHash64 | RFC 8878 frame format; full decoder (FSE + 1/4-stream Huffman); encoder: Huffman literals + predefined/RLE FSE sequences (custom sequence tables not emitted — ratio, not correctness) |
| Bzip2 | ✅ | ✅ | BWT + Huffman | CRC-32 | Block-sorting compression |
| Brotli | ✅ | ✅ | Brotli (RFC 7932) | None | Quality levels 0-11, full Appendix A static dictionary; .br file-path CLI support via extension fallback (raw Brotli has no magic bytes) |
| Snappy | ✅ | ✅ | Snappy | CRC32C | Block and framed formats |
| ISO 9660 | ✅ | ❌ | Store | None | Read-only; list/extract/info/detect support |
- AES-128 / AES-192 / AES-256 (WinZip AE-2) via a genuine, FIPS-197-compliant AES cipher (key schedule/round count derived from key length), CTR mode, HMAC-SHA1 authentication tag verified in constant time, and OS-CSPRNG-sourced salts. AE-2 entries write CRC=0 per the WinZip AES spec.
- Traditional ZipCrypto encryption/decryption, with a CSPRNG-sourced header. Info-ZIP (
zip -e) streamed archives — which derive the password-check byte from the DOS mtime rather than the CRC — decrypt correctly. - Encryption detection uses the ZIP general-purpose bit 0 (plus method 99 for AES), so archives encrypted by external tools (
zip -e, 7-Zip, WinRAR, Python) are correctly reported as encrypted and require a password — they are never silently extracted as garbage. - Spanned/multi-volume ZIP archives are not supported — both classic and Zip64 end-of-central-directory records that declare more than one disk are rejected with an explicit error rather than silently misread.
extract/list --memory-limit <BYTES> bounds memory use during decompression for every supported format. Container entries (ZIP/TAR/LZH/7z/CAB/ISO) are checked against their declared sizes before allocating. Single-file formats are enforced during decode: gzip via the trailing ISIZE field, xz via the stream index's declared uncompressed size, lz4/zstd via the frame content-size fields, and bzip2/brotli/snappy via bounded decoders (decompress_with_limit) that return an error as soon as output would exceed the limit — no pre-flight size field is required. Measured: a brotli decompression bomb extracted under --memory-limit 1M peaks at 3.3 MB RSS (vs 72.9 MB unbounded) and exits non-zero.
Independently of --memory-limit, every header-driven allocation across the readers (ZIP central directory/AES payloads, TAR PAX/extension data, LZH, 7z, ISO 9660 directory extents, zstd frame content-size, LZMA/LZMA2 dictionaries) validates the declared length against the bytes actually available and allocates via try_reserve/try_reserve_exact rather than an unconditional Vec::with_capacity/vec![0; n]. A crafted, wildly-oversized header therefore surfaces as a clean error instead of an allocator abort/OOM even with no --memory-limit set at all.
Self round-trips alone cannot prove interoperability — an encoder and decoder that share the same deviation from a spec will round-trip perfectly while being incompatible with everything else. Every OxiArc codec is therefore validated by differential tests against the reference implementation, in both directions: reference-produced streams must decode byte-identically, and oxiarc-produced streams must be accepted (and decode byte-identically) by the reference tool.
Two layers keep this permanent:
- Always-run embedded corpora — golden byte vectors generated by the reference tools are committed and checked on every
cargo nextest run, with no external dependencies. - Live oracle suites — opt-in Cargo features that shell out to the real reference tool. They self-skip with a printed note (never fail) when the tool is absent, so enabling them is always safe and CI stays hermetic.
| Crate | Feature | Reference oracle |
|---|---|---|
oxiarc-zstd |
zstd-oracle |
zstd CLI |
oxiarc-brotli |
brotli-oracle |
brotli CLI |
oxiarc-lzma |
xz-oracle |
xz (XZ Utils) CLI |
oxiarc-bzip2 |
bzip2-oracle |
bzip2 CLI |
oxiarc-lz4 |
lz4-oracle |
lz4 CLI |
oxiarc-snappy |
snappy-oracle |
python3 + cramjam |
oxiarc-deflate |
zlib-oracle |
python3 (zlib/gzip) + gzip CLI |
oxiarc-lzw |
tiff-oracle |
python3 + Pillow (libtiff), tiffcp when present |
oxiarc-szip |
libaec-oracle |
libaec (compiled harness; LIBAEC_PREFIX env var) |
oxiarc-lzhuf |
lha-oracle |
lha (Lhasa) CLI |
oxiarc-archive |
zip-oracle, xz-oracle, lha-oracle |
Info-ZIP zip/unzip + Python zipfile; xz; lha |
# Run one codec's live oracle against the reference tool
cargo nextest run -p oxiarc-zstd --features zstd-oracle
cargo nextest run -p oxiarc-brotli --features brotli-oracle
# Archive-level oracles
cargo nextest run -p oxiarc-archive --features zip-oracle,xz-oracle,lha-oracle
# Everything, everywhere (oracle suites self-skip for any missing tool)
cargo nextest run --workspace --all-featuresVerified interop snapshot (last full run 2026-07-13, live tools; unchanged in 0.4.0 — this release's DEFLATE/zlib decoder rewrite changed no wire format or output, so the DEFLATE/zlib/gzip result below still holds): zstd 64/64 corpus + 101/101 wide frames decode byte-identical, 85/85 oxiarc frames accepted by zstd -d; brotli 608/608 decode / 588/588 accepted; xz 5.8.3 60/60 decode / 8/8 encode; bzip2 1.0.8 324/324 both directions; lz4 1.10.0 11/11 + 44/44 + 3/3 linked; TIFF-LZW 125/125 vs Pillow/libtiff (encoder byte-identical to libtiff); libaec 1.1.4 2450/2450 decode + 4900/4900 encode; DEFLATE/zlib/gzip bit-exact vs CPython + gzip CLI.
Real-world performance measured on various data types:
| Level | Uniform Data | Text Data | Binary Data |
|---|---|---|---|
| Level 1 (Fast) | 400 MB/s | 85 MB/s | 48 MB/s |
| Level 5 (Normal) | 275 MB/s | 42 MB/s | 13 MB/s |
| Level 9 (Best) | 253 MB/s | 15 MB/s | 0.3 MB/s |
| Operation | Speed Range |
|---|---|
| Forward Transform | 2-11 MB/s |
| Inverse Transform | 60-320 MB/s |
| Algorithm | Naive | Slicing-by-8 | Speedup |
|---|---|---|---|
| CRC-32 | ~150 MB/s | ~500 MB/s | 3.3x |
| CRC-64 | ~100 MB/s | ~450 MB/s | 4.5x |
OxiArc implements several performance optimizations:
- CRC Slicing-by-8: Hardware-independent 3-5x speedup over table lookup
- Optimized Hash Chains: Improved LZ77 pattern matching with multiplication-based hashing
- Lazy Matching: Better compression ratios in DEFLATE with minimal speed impact
- BWT Key-Based Sorting: 4-byte prefix keys for faster block sorting
- Zero-Copy Streaming: Minimizes allocations and memory copies
- Early Rejection: Fast-path optimizations for match finding
# Create a ZIP archive from files and directories
oxiarc create backup.zip file1.txt file2.pdf documents/
# Create with compression level (store, fast, normal, best)
oxiarc create -l best archive.zip src/ tests/
# Verbose output
oxiarc create -v data.zip folder/# Create a TAR archive
oxiarc create backup.tar project/
# Combine with compression (tar.gz, tar.xz, tar.bz2, tar.zst)
gzip backup.tar # or use GZIP directly
oxiarc create backup.tar.gz folder/ # Auto-detects .gz extension# GZIP compression
oxiarc create data.txt.gz large_file.txt
# XZ (LZMA2) compression
oxiarc create database.sql.xz database.sql
oxiarc create -l best archive.xz bigdata.bin
# LZ4 (fast compression)
oxiarc create temp.lz4 file.bin
oxiarc create -l fast logs.lz4 access.log
# Zstandard compression
oxiarc create data.zst large_dataset.csv
# Bzip2 compression
oxiarc create text.bz2 document.txt# Create LZH archive (Japanese format)
oxiarc create archive.lzh file1.txt file2.txt folder/# Extract to current directory
oxiarc extract archive.zip
oxiarc extract data.tar.gz
oxiarc extract files.7z
# Extract to specific directory
oxiarc extract archive.zip -o extracted/
oxiarc extract backup.tar.xz -o /tmp/restore/
# Extract with progress bar
oxiarc extract large_archive.zip --progress
# Verbose output (show each file being extracted)
oxiarc extract data.lzh -v# Extract specific files
oxiarc extract archive.zip file1.txt readme.md
# Extract only files matching patterns (glob syntax)
oxiarc extract backup.zip --include "*.txt"
oxiarc extract data.tar --include "src/**/*.rs"
# Exclude files from extraction
oxiarc extract archive.zip --exclude "test/*" --exclude "*.tmp"
# Combine include and exclude
oxiarc extract backup.zip --include "docs/**" --exclude "*.draft"# Preserve modification timestamps
oxiarc extract archive.zip -t
# Preserve Unix file permissions
oxiarc extract backup.tar --preserve-permissions
# Preserve all metadata (timestamps + permissions)
oxiarc extract data.tar.gz -p# Always overwrite (default)
oxiarc extract archive.zip --overwrite
# Skip existing files without prompting
oxiarc extract backup.zip --skip-existing
# Prompt before overwriting each file
oxiarc extract data.zip --prompt# Decompress from stdin to stdout
cat data.gz | oxiarc extract - -o - > output.txt
curl https://example.com/data.xz | oxiarc extract - --format xz > data.txt
# Extract specific format from stdin
oxiarc extract - --format gzip < compressed.gz > original.txt# Compress to stdout
oxiarc create - --format gzip < input.txt > output.gz
cat large_file.bin | oxiarc create - --format xz > compressed.xz
# Pipe compression
find . -name "*.log" | tar -cf - -T - | oxiarc create - --format zst > logs.tar.zst# List files in archive
oxiarc list archive.zip
oxiarc list backup.tar.gz
oxiarc list data.7z
# Verbose listing (show size, date, permissions)
oxiarc list archive.zip -v
# JSON output (machine-readable)
oxiarc list data.lzh --json# List only matching files
oxiarc list backup.zip --include "*.txt"
oxiarc list archive.tar --include "src/**/*.rs"
# Exclude patterns
oxiarc list data.zip --exclude "test/*"# Test archive integrity
oxiarc test archive.zip
oxiarc test backup.tar.gz
oxiarc test data.lzh
# Verbose testing (show each file being tested)
oxiarc test archive.7z -v# Show archive metadata
oxiarc info archive.zip
oxiarc info data.7z
oxiarc info backup.lzh
# Example output:
# Format: ZIP
# Files: 42
# Compressed size: 1.2 MB
# Uncompressed size: 5.4 MB
# Compression ratio: 77.8%# Detect archive format
oxiarc detect unknown_file.bin
oxiarc detect downloaded_archive
# Useful for files without extensions
oxiarc detect mystery_file# Convert archive formats
oxiarc convert old.lzh new.zip
oxiarc convert data.7z backup.tar
oxiarc convert legacy.cab modern.zip
# Convert with compression level
oxiarc convert source.zip dest.tar -l best
# Verbose conversion
oxiarc convert old.lzh new.zip -vPattern syntax supports glob-style wildcards:
*matches any characters except/**matches any characters including/(recursive)?matches a single character[abc]matches one character from the set
# Include only specific file types
oxiarc extract archive.zip --include "*.txt" --include "*.md"
# Recursive pattern matching
oxiarc list backup.tar --include "src/**/*.rs"
oxiarc extract data.zip --include "docs/**/*.pdf"
# Complex filtering
oxiarc extract backup.zip \
--include "src/**" \
--exclude "src/test/**" \
--exclude "**/*.tmp"use oxiarc_deflate::{deflate, inflate};
use oxiarc_core::error::Result;
fn main() -> Result<()> {
// DEFLATE compression
let data = b"Hello, World! This is a test.";
let compressed = deflate(data, 6)?; // Level 6 compression
let decompressed = inflate(&compressed)?;
assert_eq!(data, &decompressed[..]);
Ok(())
}use oxiarc_archive::ZipReader;
use std::fs::File;
use std::io::Read;
fn read_zip() -> oxiarc_core::error::Result<()> {
// Open ZIP archive
let file = File::open("archive.zip")?;
let mut zip = ZipReader::new(file)?;
// List entries
for entry in zip.entries() {
println!("{}: {} bytes (compressed: {})",
entry.name,
entry.size,
entry.compressed_size
);
}
// Extract specific file
let mut data = Vec::new();
zip.extract_by_name("readme.txt", &mut data)?;
println!("Content: {}", String::from_utf8_lossy(&data));
Ok(())
}use oxiarc_archive::zip::{ZipWriter, ZipCompressionLevel};
use std::fs::File;
fn create_zip() -> oxiarc_core::error::Result<()> {
let file = File::create("output.zip")?;
let mut zip = ZipWriter::new(file);
// Add file with compression
zip.add_file(
"hello.txt",
b"Hello, World!",
ZipCompressionLevel::Normal
)?;
// Add directory
zip.add_directory("docs/")?;
// Finalize archive
zip.finish()?;
Ok(())
}use oxiarc_lzma::{compress, decompress, LzmaLevel};
fn lzma_example() -> oxiarc_core::error::Result<()> {
let data = b"This is test data for LZMA compression";
// Compress with LZMA
let compressed = compress(data, LzmaLevel::DEFAULT)?;
// Decompress
let decompressed = decompress(&compressed)?;
assert_eq!(data, &decompressed[..]);
Ok(())
}use oxiarc_bzip2::{compress, decompress, CompressionLevel};
fn bzip2_example() -> oxiarc_core::error::Result<()> {
let data = b"Data to compress with Bzip2";
// Compress (levels 1-9)
let compressed = compress(data, CompressionLevel::Best)?;
// Decompress
let decompressed = decompress(&compressed)?;
assert_eq!(data, &decompressed[..]);
Ok(())
}use oxiarc_lz4::{compress_frame, decompress_frame};
fn lz4_example() -> oxiarc_core::error::Result<()> {
let data = b"Fast compression with LZ4";
// Compress (very fast)
let compressed = compress_frame(data)?;
// Decompress
let decompressed = decompress_frame(&compressed)?;
assert_eq!(data, &decompressed[..]);
Ok(())
}use oxiarc_archive::ArchiveFormat;
use std::fs::File;
fn detect_format() -> oxiarc_core::error::Result<()> {
let mut file = File::open("unknown.bin")?;
let (format, magic) = ArchiveFormat::detect(&mut file)?;
println!("Detected format: {}", format);
println!("Magic bytes: {:02X?}", magic);
if format.is_archive() {
println!("This is a multi-file archive");
} else if format.is_compression_only() {
println!("This is single-file compression");
}
Ok(())
}# Build all crates
cargo build --release
# Run all tests (2,329 via nextest + 139 doctests = 2,468)
cargo nextest run --workspace --all-features
cargo test --doc --workspace --all-features
# Build CLI only
cargo build --release -p oxiarc-cli
# Install CLI
cargo install --path oxiarc-cli- Rust 1.85+ (Edition 2024)
- No external C libraries or compression dependencies
- Optional:
indicatiffor progress bars (CLI only)
We welcome contributions to OxiArc! Please follow these guidelines:
OxiArc is part of the COOLJAPAN ecosystem and follows strict development policies:
- No C/Fortran dependencies - All code must be pure Rust
- If C/Fortran bindings are absolutely necessary, they must be feature-gated
- Default features must be 100% pure Rust
- Code must compile with zero warnings
- Run
cargo clippyand fix all warnings before submitting - Use
cargo nextest run --all-featuresto verify
- Avoid using
.unwrap(),.expect(), or panicking code in production - Use proper error handling with
Result<T, E> - Provide meaningful error messages
- Use workspace-level dependency management
- Set
*.workspace = truein crateCargo.tomlfiles - No version specifications in individual crates (except keywords/categories)
- Always use the latest stable versions from crates.io
- Keep dependencies up to date
- Keep individual source files under 2000 lines
- Use
splitrstool for refactoring large files - Check with
rslines 50to find refactoring targets
-
Fork and Clone
git clone https://github.com/YOUR_USERNAME/oxiarc cd oxiarc -
Create a Branch
git checkout -b feature/your-feature-name
-
Make Changes
- Follow Rust naming conventions (snake_case for variables/functions)
- Add tests for new functionality
- Update documentation and examples
- Run tests:
cargo nextest run --all-features - Check code:
cargo clippy --all-features
-
Test Thoroughly
# Run all tests cargo nextest run --all-features # Check for warnings cargo clippy --all-features # Check formatting cargo fmt --check # Run benchmarks (if applicable) cargo bench
-
Commit Changes
- Write clear, descriptive commit messages
- Reference issue numbers if applicable
- DO NOT commit unless explicitly ready
- NEVER use
cargo publishwithout permission
-
Submit Pull Request
- Describe your changes clearly
- Reference related issues
- Ensure
cargo clippy --workspace --all-features --all-targetsandcargo nextest run --workspace --all-featurespass locally (this project has no CI pipeline yet, so these checks are not automated) - Wait for review from maintainers
- Follow standard Rust conventions
- Use
rustfmtfor formatting:cargo fmt - Document public APIs with doc comments (
///) - Include examples in documentation where helpful
- Prefer explicit over implicit
- Think deeply about implementations (ultrathink mode)
- Write unit tests for new functionality
- Add integration tests for complex features
- Include edge case testing
- Use temporary directories for file operations:
std::env::temp_dir() - Aim for high test coverage
- Update README.md for user-facing changes
- Update TODO.md for development progress
- Add API documentation for public items
- Include usage examples
- Keep documentation accurate and up-to-date
- Use
criterionfor benchmarks - Place benchmarks in
benches/directory - Document benchmark methodology
- Include various data patterns (uniform, random, text, binary)
When reporting issues, please include:
- Rust version (
rustc --version) - OxiArc version
- Operating system and architecture
- Minimal reproduction example
- Expected vs actual behavior
- Any relevant error messages
- Describe the use case clearly
- Explain why the feature would be useful
- Provide examples of how it would be used
- Consider implementation complexity
When adding new formats or algorithms:
- Follow the existing layered architecture
- Core algorithms go in appropriate codec crates
- Format support goes in
oxiarc-archive - CLI features go in
oxiarc-cli - Share common code through
oxiarc-core
- Be respectful and constructive
- Help others in issues and discussions
- Share knowledge and expertise
- Follow the Rust Code of Conduct
OxiARC is developed and maintained by COOLJAPAN OU (Team Kitasan).
If you find OxiARC useful, please consider sponsoring the project to support continued development of the Pure Rust ecosystem.
https://github.com/sponsors/cool-japan
Your sponsorship helps us:
- Maintain and improve the COOLJAPAN ecosystem
- Keep the entire ecosystem (OxiGDAL, OxiMedia, OxiBLAS, OxiFFT, SciRS2, etc.) 100% Pure Rust
- Provide long-term support and security updates
Licensed under the Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0).
https://github.com/cool-japan/oxiarc
COOLJAPAN OU contact@cooljapan.tech