Skip to content
RistellisePublic

About

A (reasonably) simple chunk-aligned, record-based archive format.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

6 Commits

Folders and files

Repository files navigation

Shinon ARchive (SAR)

SAR (Shinon ARchive) is a chunk-aligned, record-based archive format.

  • Pronunciation: 「さー」 (Sa) or S-A-R (like the SAR 21 rifle in Singapore). Both are officially accepted.
  • Motivation: Legacy archive formats (like TAR) carry decades of historical baggage—such as octal-string file limits, UNIX user/group ID cruft, and messy extensions. SAR aims to solve this by providing a strict 16-byte aligned, highly scalable, and developer-friendly binary format that works predictably across modern operating systems.

1. Global Format Specifications (V0)

A SAR file is a contiguous stream of Records. To ensure unambiguous and fast parsing, all compliant SAR parsers must adhere to the following global rules:

  • Chunk Alignment: The format is strictly built on 16-byte chunks.
  • Byte Order: All integer values must be specified in unsigned form (uint32) and as Big-Endian.
  • Padding: Data payloads must be padded using PKCS#7 (See RFC 5652 S6.3) to complete the final 16-byte chunk of the record. Unofficial records may implement custom padding schemas.
  • Header Requirement: A SAR file MUST begin with exactly one SAR0 - SAR9 File Header record. All subsequent records are optional.

2. Standard Header Layout

Almost all records utilize an extended 16-byte header to dictate size and integrity:

     (16 Bytes)
+----+----+----+----+
|TYPE|LENT|LENH|XXHH|
+----+----+----+----+

TYPE: Record Type Header (ascii)
LENT: No. of Chunks, Lower 32-bits (uint)
LENH: No. of Chunks, Upper 32-bits (uint)
XXHH: 4 byte hash of data payload using xxhash32 (Recommended)

3. Official Record Types

The following 4-byte ASCII identifiers are officially reserved:

  • SAR0 (Archive Header, inclusive of SAR1...SAR9)
  • SABT (Anonymous Binary / Text Record)
  • F0LE (File Record, inclusive of F1LE...F9LE)
  • F0LH / F0LS (Hard Link / Symbolic Link)
  • NEXT (Reserved for mult-part archive continuation)

3.1. SAR0 (Archive Header)

The mandatory first record of any SAR file.

+----+----+----+----+
|SAR0|LENT|LENH|XXHH|
+----+----+----+----+

SAR0: File magic (ASCII), where '0' represents the spec version.
LENT: Must be exactly 1.
LENH: Must be exactly 0.
XXHH: Set to 0x00000000 (Since there is no data payload to hash).

3.2. F0LE (File Record)

Used for typical TAR-like file storage, retaining filesystem metadata.

   (16 Bytes header) (Variable Payload)
+----+----+----+----+----+====+
|F0LE|LENT|LENH|XXHH|FILH|FILD|
+----+----+----+----+----+====+

FILH: File Header Struct (Metadata & Name)
FILD: File Data (Binary payload, PKCS#7 padded)

The FILH (File Header) Struct: The FILH immediately follows the 16-byte F0LE header. It contains the following items in strict order:

  1. PERM (1 Byte): File permissions. The lower 3 bits represent Owner permissions (Read, Write, Execute). The upper 5 bits are reserved and MUST be 0.
  2. RSVD (3 Bytes): Reserved padding bytes (0x00) to maintain 32-bit alignment.
  3. FPZZ (4 Bytes): Number of 4-byte chunks utilized by the File Path string.
  4. FPNN (Variable): The actual File Path string (UTF-8), padded to exactly FPZZ * 4 bytes.

3.3. F0LH / F0LS (Hard & Symbolic Links)

Identical in physical structure to SABT (see below), but treated as filesystem links.

  • The BLOB payload contains the UTF-8 target filepath instead of binary data.
  • Strict Rule: FPNN MUST reference a file that has already been unpacked earlier in the archive. If the target does not exist at the time of parsing, the viewer MUST throw an error, even if the target is declared later in the archive.

3.4. SABT (Anonymous Binary/Text Blob)

Used when data does NOT require file attributes (no filename, no metadata).

   (16 Bytes)        (Variable Payload)
+----+----+----+----+====+
|SABT|LENT|LENH|XXHH|BLOB|
+----+----+----+----+====+

BLOB: The raw data blob, PKCS#7 padded to the nearest 16-byte boundary.

UI/Viewer Implementation for SABT:

When listing a SAR archive, viewers SHOULD dynamically stub filenames for SABT records using a zero-indexed counter:

0000.sabt.bin
nicemovie.mkv      // Extracted from an F0LE record
0001.sabt.bin
somefile.txt       // Extracted from an F0LE record
0002.sabt.bin
...

4. Filepath Rules & Safety

To prevent malicious archive structures (e.g., directory traversal attacks) and ensure cross-platform compatibility, strict rules apply to the FPNN string.

4.1. Relative Paths Only

All filepaths MUST be relative, POSIX-style (/), and UTF-8 encoded. Other encodings (like Shift-JIS) are highly discouraged.

Valid:

  • Documents/receipts.pdf
  • Music/種島ぽぷら - SMILE SMILES.flac

Invalid (Absolute Paths & Backslashes):

  • /home/ristellise/Documents/receipts.pdf
  • C:\users\ristellise\Music\track.flac

4.2. Forbidden Characters & Names

When writing records, compliant SAR packers MUST NOT include the following:

  • Forbidden Characters: < > : " / \ | ? * %
  • Forbidden Windows Device Names: CON, PRN, AUX, NUL, COM1-COM9, LPT1-LPT9. (Note: Exact matches are forbidden. CON.txt is invalid, but CONNN.txt is valid).

Extractor Fallback: When unpacking a SAR file containing forbidden characters, the viewer/extractor SHOULD automatically replace the illegal characters with a safe fallback (implementation-defined) and emit a warning to the user regarding "tainted" or modified records.


5. Calculating LENT and LENH (Chunk Math)

SAR dynamically scales from tiny text files to exabyte-scale data blobs by combining two 32-bit unsigned integers (LENT and LENH) into a single 64-bit chunk count.

Because the SAR format relies on strict 16-byte alignment, the size of a record is defined in Chunks, where 1 Chunk = 16 Bytes.

For Parsers (Reading): When reading a record, a parser calculates the exact total chunks using the following formula: TOTAL_CHUNKS = (LENH * 4294967296) + LENT

To skip a record, the parser calculates TOTAL_CHUNKS * 16 and seeks forward by that many bytes.

For Packers (Writing): When creating a record, calculate the absolute total bytes required (including the 16-byte header, the FILH struct, the binary payload, and the PKCS#7 padding).

TOTAL_BYTES = 16 (Header) + Padded Payload
TOTAL_CHUNKS = TOTAL_BYTES / 16

Because TOTAL_CHUNKS is an exact value, packers assign LENT and LENH using standard bitwise division and modulo:

  • LENH = floor(TOTAL_CHUNKS / 4294967296)
  • LENT = TOTAL_CHUNKS % 4294967296

(Note: For 99% of files, the size will be under ~64 GiB. In this scenario, LENH will simply be 0, and LENT will contain the exact chunk count).

6. Regarding PKCS#7 Padding

All data payloads (FILD, BLOB) must be padded using PKCS#7 to perfectly align with the 16-byte chunk boundary. Because the LENT/LENH calculation is mathematically exact, the required padding will always be between 1 and 16 bytes.

For example, if a payload requires 4 bytes of padding to complete a 16-byte chunk, the packer will append 0x04 0x04 0x04 0x04 to the end of the file data.


7. Unofficial Records (Extensibility)

SAR is designed to be easily extensible. Third parties MAY declare unofficial record types, provided they do NOT use any officially reserved 4-byte ASCII headers (e.g., avoiding SAR0, F0LE).

   (16 Bytes Header)
+----+----+----+----+====+
|EXMP|LENT|LENH|... |DATA|
+----+----+----+----+====+

EXMP: A custom 4-byte string.
DATA: N bytes of custom payload.

Forward Compatibility Requirement: If a SAR parser encounters an unrecognized Record Type, it MUST NOT crash. Instead, it must read LENT and LENH, calculate the total byte size (LENT * LENH * 16), and seamlessly seek forward to skip the unknown block.


8. Out of Scope

To maintain maximum simplicity and parser performance, the following features are intentionally omitted from the core SAR specification:

  • Compression: SAR does not natively compress data. Similar to TAR, compression SHOULD be applied over the entire finalized archive using an external compressor (e.g., yielding a .sar.gz or .sar.zst file).
  • Encryption: Native encryption is not supported for official records. However, users are free to implement encryption within Unofficial Custom Records if required.

9. Implementation

This repository implements the format in Go, with Python bindings on top:

go/sar.go        Go core library: record I/O, chunk math, PKCS#7, XXH3, path safety
go/cmd/sar/      Pure-Go CLI (-c / -x / -t, same flags as the Python CLI)
go/bindings/     session layer (api.go) + cgo exports (main.go) compiled to the
                 C shared library (sar/_libsar.so)
sar/             Python package (ctypes over the shared library) + `sar` CLI
tests/           Python test suite
testdata/        Reference archive generated by the original pure-Python
                 implementation, used by the test suites for byte compatibility

Building

go build -C go -o bin/sar ./cmd/sar   # Go CLI → go/bin/sar
go test -C go ./...                   # Go tests

# Shared library → sar/_libsar.so (also builds automatically on first import)
go build -C go -buildmode=c-shared -o ../sar/_libsar.so ./bindings

uv sync             # Python package + dev environment
uv run pytest       # Python tests
uv build --wheel    # wheel (Go toolchain required at build time)

Python API

import sar

with sar.open("data.sar", "w") as sf:
    sf.add("file.bin")                          # streamed, single pass, real XXHH
    sf.addfile(sar.SARInfo("a.txt"), io.BytesIO(b"hi"))
    sf.addblob(b"raw data")                     # anonymous SABT record
    sf.addlink("bin/link", "file.bin")           # F0LS symlink record

with sar.open("data.sar", "r") as sf:
    for member in sf:                           # metadata only, no payload I/O
        ...
    sf.extractall("out/")

Thread safety

The shared library is safe to use from multiple threads, including on free-threaded (no-GIL) CPython builds. Each archive serializes its operations behind an internal mutex, the handle registry is mutex-guarded, and close() is idempotent — racing closes or post-close operations raise a clean error instead of crashing. This is verified by go test -race on the session layer (go/bindings) and by the Python thread stress test, which CI also runs on the free-threaded 3.14t build.

CI

.github/workflows/ci.yml lints and tests the Go module (with the race detector), then for each supported Python version (3.10–3.14, plus free-threaded 3.14t) builds and tests the package, builds a wheel, and smoke-tests the installed wheel. It also cross-compiles the standalone CLI for linux/darwin/windows (amd64/arm64). Pushing a v* tag attaches all wheels and binaries to a GitHub Release; publishing to PyPI is a two-line switch documented in the workflow.

About

A (reasonably) simple chunk-aligned, record-based archive format.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages