minikv

Distributed, multi-tenant key-value and object store in Rust, with Raft consensus, WAL durability, and production-oriented operations.

v2.0.0 reworks the distributed layer.

• Coordinators replicate metadata through a persistent Raft log: elections with log up-to-date checks, majority commit, automatic failover.
• Writes use two-phase commit between the leader and the volume servers, with one blob per object version.
• Reads are linearizable on every coordinator (ReadIndex).
• Volume servers run a gRPC storage service and send heartbeats to every coordinator.
• A new end-to-end test runs 3 coordinators and 3 volumes, and kills the leader along the way.

The v1.0.0 features (time-series API, vector search, Python SDK, Helm chart) are unchanged. See the CHANGELOG for details and breaking changes.

• What is minikv
• How it works
• Quick Start
• Python SDK
• Core Features
• Operations and Release Engineering
• Roadmap
• Development
• Contributing

minikv is a distributed systems reference implementation and an extensible data platform.

• Strong consistency: Raft-replicated metadata, two-phase commit to the volume servers, linearizable reads.
• Durability: write-ahead logs, checksummed segments, persistent Raft log.
• Security building blocks: API keys/JWT, RBAC, quotas, encryption.
• Real-time and analytics pathways: watch/SSE and time-series APIs.

A cluster has two kinds of nodes: coordinators, which form a Raft group and hold the metadata, and volume servers, which hold the bytes. Each volume server sends a heartbeat to every coordinator listed in --coordinators.

A write (PUT) goes through the leader:

1. The leader picks the target volumes with rendezvous hashing (HRW).
2. Prepare: each target volume receives the bytes and checks their size and BLAKE3 hash, without making them visible.
3. Commit: each volume persists the blob. If one of them fails, the write is rolled back everywhere.
4. The leader appends the new metadata to its Raft log and answers once a majority of coordinators has stored it.
5. The blob of the previous version is deleted in the background.

A read (GET) works on any coordinator:

1. The coordinator asks the leader for its commit index (ReadIndex) and waits until it has applied it.
2. It reads the metadata locally, fetches the blob from a live replica and checks its BLAKE3 hash.

If the leader goes down, the remaining coordinators elect a new one within about a second. A follower answers writes with 503 and an x-minikv-leader header that names the leader.

Build and run a local cluster of 3 coordinators and 3 volumes:

git clone https://github.com/whispem/minikv.git
cd minikv
make build
make serve

On macOS, port 5000 is taken by the AirPlay Receiver. Turn it off in System Settings › General › AirDrop & Handoff before running make serve.

Basic checks:

curl -s http://127.0.0.1:5000/health/live
curl -s http://127.0.0.1:5000/health/ready
curl -s http://127.0.0.1:5000/metrics
curl -s http://127.0.0.1:5000/admin/status

Store an object, then read it from the other coordinators. The coordinators listen on ports 5000, 5002 and 5004. Writes go to the leader: if coordinator 1 is not the leader, the PUT answers 503 and the x-minikv-leader header tells you which one is.

curl -X PUT --data-binary "hello" http://127.0.0.1:5000/s3/demo/hello.txt
curl http://127.0.0.1:5002/s3/demo/hello.txt
curl http://127.0.0.1:5004/s3/demo/hello.txt

Notebook-first SDK (preview) for data scientists and engineers:

• Time-series write/query helpers
• Vector upsert/query helpers
• SSE change-stream consumption
• Dataframe conversions (pandas, polars, pyarrow)

Install and run example:

pip install -r sdk/python/requirements.txt
python examples/data_science_quickstart.py

Files:

• sdk/python/minikv_client.py
• sdk/python/README.md
• examples/data_science_quickstart.py

Distribution and consistency:

• Raft consensus: leader election, persistent log replication, majority commit
• Two-phase commit between coordinators and volume servers, with replicated blobs checked by BLAKE3
• Linearizable reads on every coordinator (ReadIndex)
• Rendezvous hashing (HRW) placement across volume servers
• Batch and range endpoints
• Cross-DC replication primitives and conflict policies

Storage and query paths:

• Volume storage engine: append-only segments, WAL, CRC32 checksums, bloom filters, index snapshots
• RocksDB for coordinator metadata
• Time-series engine with aggregation and downsampling
• Vector similarity endpoints with cosine top-k search

Security and tenancy building blocks:

• API keys (Argon2), JWT, RBAC
• AES-256-GCM encryption
• Tenant quotas and request rate limiting
• Audit logging for admin operations

These modules are implemented and tested on their own. Enforcing them on the HTTP API is planned for v2.1.0.

APIs:

• HTTP REST and S3-compatible endpoints (PUT, GET, DELETE)
• WebSocket/SSE watch endpoints
• gRPC internal communication (Raft between coordinators, 2PC with volumes)

Observability:

• Prometheus and OpenTelemetry integration
• Grafana dashboard provisioning
• Alert rules in opentelemetry/prometheus-alerts.yml

Kubernetes:

• Operator manifests under k8s/
• Helm chart under k8s/helm/minikv/

Runbooks:

• Backup/restore: docs/ops-backup-restore.md
• Release process: docs/release-engineering-v2.0.0.md

Preflight commands:

make release-preflight
make release-preflight-full

v2.1.0:

• Authentication, RBAC, quotas and encryption enforced on the HTTP API
• Raft log compaction and snapshots
• Automatic re-replication when a volume is lost
• Write forwarding from followers to the leader
• Background compaction and cluster verify/repair tooling
• Kafka Connect sink/source templates for CDC
• Read replicas for analytical traffic
• Vector index acceleration (HNSW/PQ)
• Better analytics query ergonomics

v2.2.0:

• Dynamic cluster membership (adding and removing coordinators)
• Virtual shards driving placement and rebalancing
• Distributed transactions scope expansion
• Multi-region active-passive with explicit failover
• Point-in-time recovery (PITR)
• Policy-driven data lifecycle automation

Future:

• Multi-region active-active
• WebAssembly UDF sandbox
• Global secondary indexes
• Expanded SQL layer

make build
make test
make fmt
make clippy

End-to-end cluster test (3 coordinators, 3 volumes, leader failover):

cargo test --release --test distributed_cluster -- --nocapture

The time-series integration tests expect a coordinator on port 8000, as in CI:

cargo run --release --bin minikv-coord -- serve --id 1

Project layout:

src/
  bin/          # minikv, minikv-coord, minikv-volume
  common/       # auth, backup, cdc, metrics, replication, timeseries, ...
  coordinator/  # Raft, metadata, placement, HTTP/gRPC APIs
  volume/       # volume node storage and APIs
  ops/          # integrity, compact, repair tooling
k8s/            # operator manifests and Helm chart
opentelemetry/  # Prometheus/Grafana/Jaeger stack
sdk/python/     # notebook-first Python client preview
docs/           # runbooks and release engineering docs

Contributions are welcome. See CONTRIBUTING.md for coding, testing, and PR workflow.

MIT. See LICENSE.