SAR (Shinon ARchive) is a chunk-aligned, record-based archive format.
- Pronunciation: 「さー」 (Sa) or S-A-R (like the SAR 21 rifle in Singapore). Both are officially accepted.
- Motivation: Legacy archive formats (like TAR) carry decades of historical baggage—such as octal-string file limits, UNIX user/group ID cruft, and messy extensions. SAR aims to solve this by providing a strict 16-byte aligned, highly scalable, and developer-friendly binary format that works predictably across modern operating systems.
A SAR file is a contiguous stream of Records. To ensure unambiguous and fast parsing, all compliant SAR parsers must adhere to the following global rules:
- Chunk Alignment: The format is strictly built on 16-byte chunks.
- Byte Order: All integer values must be specified in unsigned form (
uint32) and as Big-Endian. - Padding: Data payloads must be padded using PKCS#7 (See RFC 5652 S6.3) to complete the final 16-byte chunk of the record. Unofficial records may implement custom padding schemas.
- Header Requirement: A SAR file MUST begin with exactly one
SAR0-SAR9File Header record. All subsequent records are optional.
Almost all records utilize an extended 16-byte header to dictate size and integrity:
(16 Bytes)
+----+----+----+----+
|TYPE|LENT|LENH|XXHH|
+----+----+----+----+
TYPE: Record Type Header (ascii)
LENT: No. of Chunks, Lower 32-bits (uint)
LENH: No. of Chunks, Upper 32-bits (uint)
XXHH: 4 byte hash of data payload using xxhash32 (Recommended)
The following 4-byte ASCII identifiers are officially reserved:
SAR0(Archive Header, inclusive ofSAR1...SAR9)SABT(Anonymous Binary / Text Record)F0LE(File Record, inclusive ofF1LE...F9LE)F0LH/F0LS(Hard Link / Symbolic Link)NEXT(Reserved for mult-part archive continuation)
The mandatory first record of any SAR file.
+----+----+----+----+
|SAR0|LENT|LENH|XXHH|
+----+----+----+----+
SAR0: File magic (ASCII), where '0' represents the spec version.
LENT: Must be exactly 1.
LENH: Must be exactly 0.
XXHH: Set to 0x00000000 (Since there is no data payload to hash).
Used for typical TAR-like file storage, retaining filesystem metadata.
(16 Bytes header) (Variable Payload)
+----+----+----+----+----+====+
|F0LE|LENT|LENH|XXHH|FILH|FILD|
+----+----+----+----+----+====+
FILH: File Header Struct (Metadata & Name)
FILD: File Data (Binary payload, PKCS#7 padded)
The FILH (File Header) Struct:
The FILH immediately follows the 16-byte F0LE header. It contains the following items in strict order:
- PERM (1 Byte): File permissions. The lower 3 bits represent Owner permissions (Read, Write, Execute). The upper 5 bits are reserved and MUST be
0. - RSVD (3 Bytes): Reserved padding bytes (0x00) to maintain 32-bit alignment.
- FPZZ (4 Bytes): Number of 4-byte chunks utilized by the File Path string.
- FPNN (Variable): The actual File Path string (UTF-8), padded to exactly
FPZZ * 4bytes.
Identical in physical structure to SABT (see below), but treated as filesystem links.
- The
BLOBpayload contains the UTF-8 target filepath instead of binary data. - Strict Rule:
FPNNMUST reference a file that has already been unpacked earlier in the archive. If the target does not exist at the time of parsing, the viewer MUST throw an error, even if the target is declared later in the archive.
Used when data does NOT require file attributes (no filename, no metadata).
(16 Bytes) (Variable Payload)
+----+----+----+----+====+
|SABT|LENT|LENH|XXHH|BLOB|
+----+----+----+----+====+
BLOB: The raw data blob, PKCS#7 padded to the nearest 16-byte boundary.
When listing a SAR archive, viewers SHOULD dynamically stub filenames for SABT records using a zero-indexed counter:
0000.sabt.bin
nicemovie.mkv // Extracted from an F0LE record
0001.sabt.bin
somefile.txt // Extracted from an F0LE record
0002.sabt.bin
...
To prevent malicious archive structures (e.g., directory traversal attacks) and ensure cross-platform compatibility, strict rules apply to the FPNN string.
All filepaths MUST be relative, POSIX-style (/), and UTF-8 encoded. Other encodings (like Shift-JIS) are highly discouraged.
Valid:
Documents/receipts.pdfMusic/種島ぽぷら - SMILE SMILES.flac
Invalid (Absolute Paths & Backslashes):
/home/ristellise/Documents/receipts.pdfC:\users\ristellise\Music\track.flac
When writing records, compliant SAR packers MUST NOT include the following:
- Forbidden Characters:
< > : " / \ | ? * % - Forbidden Windows Device Names:
CON,PRN,AUX,NUL,COM1-COM9,LPT1-LPT9. (Note: Exact matches are forbidden.CON.txtis invalid, butCONNN.txtis valid).
Extractor Fallback: When unpacking a SAR file containing forbidden characters, the viewer/extractor SHOULD automatically replace the illegal characters with a safe fallback (implementation-defined) and emit a warning to the user regarding "tainted" or modified records.
SAR dynamically scales from tiny text files to exabyte-scale data blobs by combining two 32-bit unsigned integers (LENT and LENH) into a single 64-bit chunk count.
Because the SAR format relies on strict 16-byte alignment, the size of a record is defined in Chunks, where 1 Chunk = 16 Bytes.
For Parsers (Reading):
When reading a record, a parser calculates the exact total chunks using the following formula:
TOTAL_CHUNKS = (LENH * 4294967296) + LENT
To skip a record, the parser calculates TOTAL_CHUNKS * 16 and seeks forward by that many bytes.
For Packers (Writing):
When creating a record, calculate the absolute total bytes required (including the 16-byte header, the FILH struct, the binary payload, and the PKCS#7 padding).
TOTAL_BYTES = 16 (Header) + Padded Payload
TOTAL_CHUNKS = TOTAL_BYTES / 16
Because TOTAL_CHUNKS is an exact value, packers assign LENT and LENH using standard bitwise division and modulo:
LENH = floor(TOTAL_CHUNKS / 4294967296)LENT = TOTAL_CHUNKS % 4294967296
(Note: For 99% of files, the size will be under ~64 GiB. In this scenario, LENH will simply be 0, and LENT will contain the exact chunk count).
All data payloads (FILD, BLOB) must be padded using PKCS#7 to perfectly align with the 16-byte chunk boundary. Because the LENT/LENH calculation is mathematically exact, the required padding will always be between 1 and 16 bytes.
For example, if a payload requires 4 bytes of padding to complete a 16-byte chunk, the packer will append 0x04 0x04 0x04 0x04 to the end of the file data.
SAR is designed to be easily extensible. Third parties MAY declare unofficial record types, provided they do NOT use any officially reserved 4-byte ASCII headers (e.g., avoiding SAR0, F0LE).
(16 Bytes Header)
+----+----+----+----+====+
|EXMP|LENT|LENH|... |DATA|
+----+----+----+----+====+
EXMP: A custom 4-byte string.
DATA: N bytes of custom payload.
Forward Compatibility Requirement:
If a SAR parser encounters an unrecognized Record Type, it MUST NOT crash. Instead, it must read LENT and LENH, calculate the total byte size (LENT * LENH * 16), and seamlessly seek forward to skip the unknown block.
To maintain maximum simplicity and parser performance, the following features are intentionally omitted from the core SAR specification:
- Compression: SAR does not natively compress data. Similar to TAR, compression SHOULD be applied over the entire finalized archive using an external compressor (e.g., yielding a
.sar.gzor.sar.zstfile). - Encryption: Native encryption is not supported for official records. However, users are free to implement encryption within Unofficial Custom Records if required.
This repository implements the format in Go, with Python bindings on top:
go/sar.go Go core library: record I/O, chunk math, PKCS#7, XXH3, path safety
go/cmd/sar/ Pure-Go CLI (-c / -x / -t, same flags as the Python CLI)
go/bindings/ session layer (api.go) + cgo exports (main.go) compiled to the
C shared library (sar/_libsar.so)
sar/ Python package (ctypes over the shared library) + `sar` CLI
tests/ Python test suite
testdata/ Reference archive generated by the original pure-Python
implementation, used by the test suites for byte compatibility
go build -C go -o bin/sar ./cmd/sar # Go CLI → go/bin/sar
go test -C go ./... # Go tests
# Shared library → sar/_libsar.so (also builds automatically on first import)
go build -C go -buildmode=c-shared -o ../sar/_libsar.so ./bindings
uv sync # Python package + dev environment
uv run pytest # Python tests
uv build --wheel # wheel (Go toolchain required at build time)import sar
with sar.open("data.sar", "w") as sf:
sf.add("file.bin") # streamed, single pass, real XXHH
sf.addfile(sar.SARInfo("a.txt"), io.BytesIO(b"hi"))
sf.addblob(b"raw data") # anonymous SABT record
sf.addlink("bin/link", "file.bin") # F0LS symlink record
with sar.open("data.sar", "r") as sf:
for member in sf: # metadata only, no payload I/O
...
sf.extractall("out/")The shared library is safe to use from multiple threads, including on free-threaded
(no-GIL) CPython builds. Each archive serializes its operations behind an internal
mutex, the handle registry is mutex-guarded, and close() is idempotent — racing
closes or post-close operations raise a clean error instead of crashing. This is
verified by go test -race on the session layer (go/bindings) and by the Python
thread stress test, which CI also runs on the free-threaded 3.14t build.
.github/workflows/ci.yml lints and tests the Go module (with the race
detector), then for each supported Python version (3.10–3.14, plus free-threaded 3.14t) builds and
tests the package, builds a wheel, and smoke-tests the installed wheel. It also cross-compiles the
standalone CLI for linux/darwin/windows (amd64/arm64). Pushing a v* tag attaches all wheels and
binaries to a GitHub Release; publishing to PyPI is a two-line switch documented in the workflow.