tqdb

The quantization-native vector database. Pure Go. Single file. Embeddable.

Store vectors compressed. Search without decompressing. Open in milliseconds.

Built on Google's TurboQuant (ICLR 2026) with Hadamard rotation, HNSW graph indexing, and NEON-accelerated distance kernels.

# Library
go get github.com/scotteveritt/tqdb

# CLI
go install github.com/scotteveritt/tqdb/cmd/tqdb@latest

# Initialize a workspace with an embedding provider
tqdb init --provider ollama

# Import data (JSONL with vectors, or --embed to auto-embed text)
tqdb import --from embeddings.jsonl

# Search by text (embeds via configured provider)
tqdb search "how does authentication work"

# Search with filters
tqdb search "error handling" --top 5 --filter repo=myrepo --filter language=go

# Inspect
tqdb info
tqdb count
tqdb bench --queries 100
tqdb export | head -5

The CLI supports three embedding providers (Vertex AI, OpenAI, Ollama) as lightweight HTTP clients with no SDK dependencies. Configure once via tqdb init or ~/.config/tqdb/config.yaml.

import (
    "github.com/scotteveritt/tqdb"
    "github.com/scotteveritt/tqdb/store"
)

// Create and populate a .tq file
s, _ := store.Create("index.tq", tqdb.StoreConfig{
    Dim: 3072, Bits: 8, Rotation: tqdb.RotationHadamard,
})
s.Add(tqdb.Document{
    ID: "doc-1", Content: "hello world", Embedding: vec,
    Data: map[string]any{"repo": "myrepo", "language": "go"},
})
s.Close() // writes the .tq file atomically

// Open (mmap, instant) and search
s, _ = store.Open("index.tq")
defer s.Close()
results := s.SearchWithOptions(query, tqdb.SearchOptions{
    TopK:   10,
    Filter: tqdb.And(tqdb.Eq("repo", "myrepo"), tqdb.Gt("stars", 10.0)),
})

coll, _ := store.NewCollection(tqdb.Config{
    Dim: 3072, Bits: 8, Rotation: tqdb.RotationHadamard,
})
coll.Add("id", vec, data)

// Build index (auto-selects IVF for N >= 10K, brute-force otherwise)
coll.CreateIndex(tqdb.IndexConfig{
    FilterFields: []string{"repo", "language"},
    // Type: tqdb.IndexAuto (default), tqdb.IndexIVF, tqdb.IndexHNSW, tqdb.IndexNone
})

1. Normalize the vector to unit length, store the magnitude separately
2. Rotate via Randomized Walsh-Hadamard Transform (O(d log d), 65 KB memory)
3. Quantize each coordinate with a Lloyd-Max codebook precomputed from the known Gaussian distribution (no training data needed)
4. Index via HNSW graph for sub-linear search, or brute-force for small collections
5. Search by rotating the query once, then traversing the graph with NEON-accelerated distance (no decompression)

The codebook depends only on (dimension, bits), not on your data. This makes quantization data-oblivious: you can add vectors one at a time without retraining.

All measurements on Apple M4 Pro with NEON acceleration.

d=128, 8-bit:

d=768, 8-bit:

IVF wins at most scales. HNSW catches up at 100K+ where O(log N) beats O(sqrt(N)). Auto mode (IndexAuto, the default) selects IVF for N >= 10K, brute-force below.

GoAT-generated ARM64 NEON kernels for distance computation:

On x86, pure Go fallbacks are used automatically.

Default is 8-bit. Indices are bit-packed in the .tq format (4-bit stores 2 indices per byte).

TurboQuant works best on modern learned embeddings (Gemini, GloVe, OpenAI).

Composable filters matching Google Vector Search 2.0 syntax:

tqdb.Eq("repo", "tqdb")
tqdb.In("lang", "go", "rust", "python")
tqdb.Gt("stars", 100.0)
tqdb.And(filter1, filter2)
tqdb.Or(filter1, filter2)
tqdb.Contains("content", "vector")

// Collection (in-memory, supports all operations)
coll.Add("id", vec, data)                   // skip if duplicate
coll.Upsert("id", vec, data)               // replace if exists
coll.Delete("id-1", "id-2")
coll.AddDocument(ctx, doc)                  // auto-embed via EmbedFunc
doc, ok := coll.GetByID("id")

// Store (file-backed, write-once)
s.Add(tqdb.Document{ID: "id", Embedding: vec, Content: "...", Data: data})

The .tq format is a single columnar file, memory-mapped for instant startup:

[Header 64B] [Indices] [Norms] [IDs] [Data] [Contents] [HNSW Graph]

Indices are bit-packed. The HNSW graph section is optional (~134 bytes/node). IDs, metadata, and content are lazily loaded on first access.

MIT