Metarc

Compress structure before bytes.

Metarc is an experimental archive format for source-code trees and structured file collections. It exploits cross-file and semantic redundancy before applying zstd.

On my personal 6.5G source-code directory: 1.4G with Metarc vs 1.8G with tar+zstd (about 22% smaller).

On well-known open-source repositories, current benchmarks show source trees 4% to 19% smaller than tar+zstd, measured on git archive exports of pinned commits: the only form that reproduces byte for byte. Archiving a full clone gains less, from roughly parity to about 8%, because .git packfiles are already compressed and neither tool can do anything with them.

Not a tar replacement yet. A research-grade playground with reproducible benchmarks.

On real-world datasets like your Go module cache, Metarc typically achieves modest but consistent gains.

# Check your GOMODCACHE size
du -sh $(go env GOMODCACHE)

# Clone the repo
git clone https://github.com/arhuman/metarc-go.git

# Install marc
cd metarc-go
make install

# Compress with tar+zstd
tar --zstd -cf /tmp/gomodcache.tar.zst -C $(go env GOMODCACHE) .

# Compress with Metarc 
marc archive /tmp/gomodcache.marc $(go env GOMODCACHE)

You can now check the results

ls -lh /tmp/gomodcache.*
perl -e 'printf "marc archive is %.2f%% smaller than tar archive\n", 100 * (1 - (-s "/tmp/gomodcache.marc") / (-s "/tmp/gomodcache.tar.zst"))'

Traditional compressors (like gzip, zstd) operate on byte streams.

Metarc explores a different idea:

compress meaning first, bytes second

Instead of only compressing raw data, it tries to:

• deduplicate repeated content across files
• normalize structured formats (JSON, logs, etc.)
• detect common patterns (licenses, boilerplate, generated code)

Then it applies a standard compressor on top.

The goal is to unlock optimizations that byte-level compression alone cannot see.

Metarc is experimental, but already usable.

• Works on real repositories
• Supports multiple transforms and strategies
• Designed for experimentation and iteration

Metarc is not (yet) trying to replace tar.

It exists to explore a different space:

• cross-file compression
• semantic transforms
• corpus-aware optimization
• new compression heuristics

Think of it as a playground for compression ideas, not a finished product.

More detailed benchmarks, as well as instructions to reproduce them, are available in docs/benchmarks.md

Metarc compression shines in directories with a lot of redundancy, where its file deduplication can outperform even tar+zstd:

6.5G	code_perso
1.8G	code_perso.tar.zst
1.4G	code_perso.marc     (22% smaller)

But the goal is to make it at least "as good" in most common cases, that's why we mainly use popular open-source repositories (using various languages) to measure our progress in this area.

Previous comparisons used tar+gzip, we now use tar+zstd for a fairer comparison.

Each repository is measured in two forms, because one number cannot answer both questions worth asking:

• full clone, .git included: what a user actually gets when archiving a working directory.
• source tree only, exported with git archive at the pinned commit: what the compressor actually did.

The gap between the two is the weight of the packfiles, which are already deflate-compressed and pass through both tools untouched. On kubernetes .git is 12% of the input bytes but roughly 58% of the resulting archive.

./scripts/run_bench.sh --type size

_marc: metarc version v0.11.1 (c87305c1, 2026-08-14T11:49:32Z) | tar: bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 _

Treat this table as indicative, not reproducible. A shallow clone does not repack byte-identically between fetches, so the tar baseline drifts on its own: the previous run of these same pinned commits measured kubernetes at 81.0M and express at 345.6K against the 79.2M and 329.8K above, while marc's output was identical to the byte in both runs. docker-compose crossing 100% comes from that baseline side too: marc wrote the same bytes, tar's shrank.

./scripts/run_bench.sh --type size --corpus tree

_marc: metarc version v0.11.1 (c87305c1, 2026-08-14T11:49:32Z) | tar: bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 _

Rows sorted by advantage. This table is byte-reproducible: two builds hours apart (v0.10.0-15 and v0.11.1) produced these figures identically. bootstrap gains the least from dropping .git (2.0 points) because its source tree is itself full of incompressible content (fonts, images, minified bundles), so removing the packfiles unmasks little. Corpora rank by how much compressible text they hold.

See docs/benchmarks.md for the gz baseline, time benchmarks, methodology, reproducibility notes, and changelog.

Metarc optimizes for size, not for archive wall-clock.

./scripts/run_bench.sh --type time

_marc: metarc version v0.11.1 (c87305c1, 2026-08-14T11:49:32Z) | tar: bsdtar 3.5.3 - libarchive 3.7.4 zlib/1.2.12 liblzma/5.4.3 bz2lib/1.0.8 _ _host: 26.5.2, Apple M3 Pro, 12 cores, 36G | timing: median of 3 runs, cache flushed before each run (cold) _

Archiving is slower on the four largest corpora and faster on the three smallest, with redis at parity. Extraction is faster on six out of eight.

Against tar+gz the same binary is faster on all eight (1.1× to 2.9×, table in docs/benchmarks.md). marc's own times are identical in both comparisons; only the baseline changes. It sits between gzip and zstd in CPU cost while producing smaller archives than either.

That marc is slower at all is a deliberate default, not a regression: solid blocks are compressed at zstd level 11. On react, measured in two alternating passes so neither ordering nor warmup can explain the gap:

Level 3 archives faster than tar+zstd and compresses like it; level 11 costs 1.6× the time and buys 6.2 points of ratio. Both sizes reproduced exactly across the two passes. --zstd-level-solid 7 sits between the two, keeping about half the ratio gain for roughly half the CPU.

git clone https://github.com/arhuman/metarc-go
cd metarc-go
make install

This installs marc to your $GOBIN (or $GOPATH/bin).

make test

marc archive repo.marc ./my-repo

marc extract repo.marc --dest restored/

marc inspect repo.marc

marc bench ./my-repo

• docs/metacompression.md conceptual background
• docs/architecture.md format, pipeline, transforms
• docs/benchmarks.md benchmarks

• ⭐ Star this repo if you find it useful
• 🐛 Report a bug
• 💡 Suggest a feature
• 🔧 Propose a transform

MIT -- see LICENSE.