LZAV is a fast, general-purpose, in-memory data compression algorithm based on the classic LZ77 lossless compression method. LZAV holds a strong position on the Pareto frontier among many similar in-memory (non-streaming) compression algorithms.

LZAV's code is portable, cross-platform, scalar, and header-only. It is also compatible with C++. It supports big- and little-endian platforms and any memory alignment model. The algorithm is efficient on both 32- and 64-bit platforms. It is compatible with WebAssembly and, when compiled against WASI libc, achieves roughly half the speed of native code. The algorithm does not measurably increase the size of incompressible data.

LZAV performs internal out-of-bounds (OOB) checks and does not trade safety for decompression speed. This means that LZAV can be used under strict conditions where OOB memory reads and writes, which could lead to CPU traps, are unacceptable (e.g., in real-time, system-level, and server software). LZAV can be used safely (without crashes or undefined behavior) even when decompressing malformed or damaged data. Consequently, it does not require calculating a checksum (or a hash) of the compressed data. Only a checksum of the uncompressed data may be required, depending on the application's needs.

The internal functions in lzav.h allow developers to implement and test their own compression algorithms. LZAV's stream format allows for high compression ratios and high decompression speeds.

To compress data:

#include "lzav.h"

int max_len = lzav_compress_bound( src_len );
void* comp_buf = malloc( max_len );
int comp_len = lzav_compress_default( src_buf, comp_buf, src_len, max_len );

if( comp_len == 0 && src_len != 0 )
{
    // Error handling
}

To decompress data:

#include "lzav.h"

void* decomp_buf = malloc( src_len );
int l = lzav_decompress( comp_buf, decomp_buf, comp_len, src_len );

if( l < 0 )
{
    // Error handling
}

To compress data with a higher compression ratio for non-time-critical uses (e.g., compression of an application's static assets):

#include "lzav.h"

int max_len = lzav_compress_bound_hi( src_len ); // Note the different bound function
void* comp_buf = malloc( max_len );
int comp_len = lzav_compress_hi( src_buf, comp_buf, src_len, max_len );

if( comp_len == 0 && src_len != 0 )
{
    // Error handling
}

LZAV's source code conforms to ISO C99 and has been tested with Clang, GCC, MSVC, and the Intel C++ compiler on x86, x86-64 (Intel, AMD), and AArch64 (Apple Silicon) systems running Windows 10, AlmaLinux 9.3, and macOS 15.7. Full C++ compatibility is automatically provided when the source code is compiled with a C++ compiler.

• C#, by AQtun
• C++, via vcpkg
• FreeArc, by Shegorat
• Rust, by pkolaczk

In C++ environments where it is undesirable to place LZAV symbols in the global namespace, the LZAV_NS_CUSTOM macro can be defined externally:

#define LZAV_NS_CUSTOM lzav
#include "lzav.h"

For example, to place LZAV symbols in a custom namespace alongside your other data compression functions:

#define LZAV_NS_CUSTOM my_compressors
#include "lzav.h"

As a result, LZAV functions can be referred to as my_compressors::lzav_compress_default(...). Note that because all LZAV functions are declared with the static specifier, there will be no ABI conflicts, even if the lzav.h header is included in multiple C/C++ translation units.

The tables below present ballpark performance numbers for the LZAV algorithm.

While LZ4 compresses faster, LZAV provides 16% greater storage space savings. This is a significant benefit in database and filesystem use cases, since LZAV's compression is only about 30% slower than LZ4's. In practice, slower compression is not a limiting factor because writes of compressed data are deferred to background threads, and disk I/O time is reduced due to better compression. In general, LZAV holds a very strong position in this class of data compression algorithms when one considers all factors: compression and decompression speeds, compression ratio, and, just as importantly, code maintainability. LZAV is highly portable and has a rather small, independent codebase.

The performance of LZAV is not limited to the ballpark numbers presented. Depending on the data being compressed, LZAV can achieve compression speeds of 800 MB/s and decompression speeds of 5,000 MB/s. Incompressible data is decompressed at a rate of 10,000 MB/s, which is close to the speed of memcpy(). For datasets such as enwik9, LZAV provides 22% greater storage space savings than LZ4.

The geometric mean speeds of the LZAV algorithm across a variety of datasets are 550 ± 150 MB/s for compression and 3,800 ± 1,300 MB/s for decompression. These numbers apply to 64-bit processors released since 2019 that run at 4 GHz or higher. Note that the algorithm exhibits adaptive qualities, and its actual performance depends on the data being compressed. LZAV may show exceptional performance on your specific data, including, but not limited to, sparse databases, log files, and HTML/XML files.

It is also worth noting that compression methods like LZAV and LZ4 usually have an advantage over explicit dictionary- and entropy-based methods. Hash-table-based compression has low memory usage and minimal operational overhead, and classic LZ77 decompression has no overhead at all; this is especially relevant for smaller datasets.

For a more comprehensive benchmark of in-memory compression algorithms, see lzbench.

The benchmarks below use the Silesia compression corpus.

Note: The popular Zstd compressor is not included here because it is not a pure LZ77 algorithm, it is much harder to integrate, and it has a much larger codebase. These aspects place it in the same league as zlib. Here are the author's Zstd measurements with TurboBench on a Ryzen 3700X, using the Silesia dataset:

This section presents compression ratio comparisons for various popular datasets. Note that each file within these datasets was compressed individually, which is reflected in the overall ratio.

1. The LZAV API is not equivalent to the LZ4 and Snappy APIs. For example, the dstlen parameter of the decompression function must specify the original uncompressed length, which must have been previously stored in some way, independently of LZAV.

2. From a technical point of view, LZAV's peak decompression speed is implicitly limited by its stream format, which is more complex than LZ4's. LZAV decompression requires extensive branching. Another limiting factor is a rather large dynamic LZ77 window (2 MiB to 512 MiB), which is not CPU-cache-friendly. However, without these features, it would not be possible to achieve competitive compression ratios while maintaining high compression speeds.

3. LZAV supports compression of data blocks up to 2 GiB. Larger datasets should be compressed in chunks of at least 16 MiB. Using smaller chunks may reduce the compression ratio.