Skip to content
gregnazarioPublic

About

Immutable indexed binary file format for time-series range queries, with native implementations in Rust, Go, Python, and TypeScript

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

18 Commits

Folders and files

Repository files navigation

Golf

License: Apache 2.0

A read-only indexed binary file format optimized for range queries over time-series keys with small fixed-size record values.

Golf files are created by appending records and then sealing the file, after which they are immutable and efficient for random range reads.

Key Features

  • Range queries via binary search over a block index -- O(log B + log N)
  • Block-level compression (LZ4 or Zstd) -- only touched blocks are decompressed
  • mmap + standard I/O read strategies
  • Fixed-size records for O(1) random access within decompressed blocks
  • CRC32C checksums at header, block, and index levels
  • Configurable timestamp resolution (nanoseconds, microseconds, milliseconds)
  • Multi-language: Rust, Go, Python, TypeScript, Swift, Kotlin

Language Support

Package names below are local module names, not published packages -- nothing is on crates.io, npm, PyPI, or any Go module registry yet; build from source (see Installation).

Language Package / Module Write Read (LZ4) Zstd In-memory Reader mmap Reader
Rust golf crate ✅ ✅ ✅ ✅ ✅
Go golf-format/golf-go ✅ ✅ ✅ ✅ (bytes.Reader) ✅
Python golf-format ✅ ✅ ✅ ✅ ✅
TypeScript golf-format (npm) ✅ ✅ ❌¹ ✅ n/a²
Swift Golf package ✅ ✅ ❌¹ ✅ planned
Kotlin/JVM golf package ✅ ✅ ❌¹ ✅ planned
  1. These implementations raise a clear error when encountering Zstd instead of bundling a decoder; see docs/COMPATIBILITY.md.
  2. The TypeScript reader is buffer-based, which plays the same role as mmap on the JS runtime.

All six implementations are byte-compatible with each other and validated against shared fixtures on every test run -- a file written by any language can be read by every language that supports its codec.

File Format

See SPEC.md for the complete byte-level specification.

┌──────────────────────┐  offset 0
│       Header         │  64 bytes + optional metadata
├──────────────────────┤
│    Data Block 0      │  compressed
│    Data Block 1      │
│       ...            │
│    Data Block N-1    │
├──────────────────────┤
│     Block Index      │  40 bytes per block descriptor
├──────────────────────┤
│       Footer         │  32 bytes (at EOF)
└──────────────────────┘

Quick Start

Rust

use golf::{GolfWriter, GolfReader};
use golf::writer::WriterConfig;
use golf::header::{Compression, TimestampResolution};

// Write
let mut writer = GolfWriter::new(WriterConfig {
    record_value_size: 8,
    compression: Compression::Lz4,
    block_capacity: 8192,
    ..Default::default()
});
writer.append(1000, vec![1u8; 8]).unwrap();
writer.append(2000, vec![2u8; 8]).unwrap();
let file = std::fs::File::create("data.golf").unwrap();
writer.seal(file).unwrap();

// Read
let mut reader = GolfReader::open_path("data.golf").unwrap();
let records = reader.query(1000, 2000).unwrap();

More: rust/README.md

Go

import golf "github.com/golf-format/golf-go"

// Write
w := golf.NewWriter(golf.WriterConfig{
    RecordValueSize: 8,
    Compression:     golf.CompressionLz4,
    BlockCapacity:   8192,
})
w.Append(1000, []byte{1,1,1,1,1,1,1,1})
w.Append(2000, []byte{2,2,2,2,2,2,2,2})
f, _ := os.Create("data.golf")
w.Seal(f)

// Read
reader, _ := golf.OpenFile("data.golf")
records, _ := reader.Query(1000, 2000)

More: go/README.md

Python

from golf import GolfWriter, GolfReader, WriterConfig, Compression

# Write
writer = GolfWriter(WriterConfig(record_value_size=8, compression=Compression.LZ4))
writer.append(1000, b'\x01' * 8)
writer.append(2000, b'\x02' * 8)
with open('data.golf', 'wb') as f:
    writer.seal(f)

# Read
with GolfReader.open('data.golf') as reader:
    records = reader.query(1000, 2000)

Install from this repo: pip install ./python. More: python/README.md

TypeScript (Node.js)

import { GolfWriter, GolfReader, Compression } from 'golf-format';

// Write
const writer = new GolfWriter({ recordValueSize: 8, compression: Compression.Lz4 });
writer.append(1000n, Buffer.alloc(8, 1));
writer.append(2000n, Buffer.alloc(8, 2));
fs.writeFileSync('data.golf', writer.seal());

// Read
const reader = GolfReader.open('data.golf');
const records = reader.query(1000n, 2000n);

Timestamps are JavaScript BigInt. More: typescript/README.md

Swift

import Golf

// Write
let writer = GolfWriter(config: WriterConfig(recordValueSize: 8, compression: .lz4))
try writer.append(timestamp: 1000, value: [UInt8](repeating: 1, count: 8))
try writer.append(timestamp: 2000, value: [UInt8](repeating: 2, count: 8))
try writer.seal(to: URL(fileURLWithPath: "data.golf"))

// Read
let reader = try GolfReader.open("data.golf")
let records = try reader.query(start: 1000, end: 2000)

Zero third-party dependencies (software CRC32C + built-in LZ4 block codec). More: swift/README.md

Kotlin

import golf.*

// Write
val writer = GolfWriter(WriterConfig(recordValueSize = 8, compression = Compression.LZ4))
writer.append(1000UL, ByteArray(8) { 1 })
writer.append(2000UL, ByteArray(8) { 2 })
writer.sealTo(File("data.golf"))

// Read
val reader = GolfReader.open("data.golf")
val records = reader.query(1000UL, 2000UL)

Timestamps are ULong; zero third-party dependencies (JDK CRC32C + built-in LZ4 block codec). More: kotlin/README.md

Installation

Golf packages are not yet published -- there is nothing on crates.io, PyPI, npm, or any Go module registry. For now, install from source and verify each implementation by running its test suite:

git clone https://github.com/gregnazario/golf.git
cd golf

# Rust
cd rust && cargo test

# Go
cd ../go && go test ./...

# Python
cd ../python
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]" && pytest

# TypeScript
cd ../typescript && npm install && npm run build && npm test

# Swift
cd ../swift && swift test

# Kotlin (uses kotlinc directly -- no Gradle/network needed)
bash scripts/test.sh

A passing test suite means the implementation is built and usable locally from its directory (e.g., import golf from the activated Python virtualenv, or import from typescript/dist/ after building). Each language directory has its own README with package-level install instructions. See Running Tests for details, including the full cross-language compatibility suite.

Running Tests

# Rust
cd rust && cargo test

# Go
cd go && go test ./...

# Python
cd python && python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]" && pytest

# TypeScript
cd typescript && npm install && npm test

# Swift
cd swift && swift test

# Kotlin (uses kotlinc directly -- no Gradle/network needed)
bash kotlin/scripts/test.sh

# Full cross-language compatibility suite (writes fixtures from every
# implementation, then verifies every implementation reads them all)
bash testdata/run_compat_tests.sh

The compat suite regenerates everything under testdata/*.golf; see docs/COMPATIBILITY.md for what each fixture exercises.

Documentation Index

Document Contents
SPEC.md Canonical byte-level file format specification
docs/COMPATIBILITY.md Cross-language parity guarantees + fixture suite
CONTRIBUTING.md How to change the format/implementations safely
rust/README.md, go/README.md, ... Per-language install, API reference, examples

Project Structure

golf/
  SPEC.md              -- Canonical binary format specification
  rust/                -- Rust implementation (reference)
  go/                  -- Go implementation
  python/              -- Python implementation
  typescript/          -- TypeScript/Node.js implementation
  swift/               -- Swift package (zero dependencies)
  kotlin/              -- Kotlin/JVM library (zero dependencies)
  testdata/            -- Cross-language test fixtures + compat runner
  docs/                -- Design & compatibility documentation

License

Apache-2.0 -- see LICENSE.

About

Immutable indexed binary file format for time-series range queries, with native implementations in Rust, Go, Python, and TypeScript

Topics

Resources

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages