A production-ready, segmented key-value storage engine built in Rust

Features • Quick Start • Architecture • API Documentation • Benchmarks • Contributing

Mini KV Store v2 is a high-performance, append-only key-value storage engine with HTTP API capabilities.
Built as an educational project to explore storage engine fundamentals, it implements core database concepts like segmented logs, compaction, bloom filters, index snapshots, and crash recovery.

💡 New: 3-week learning journey — from Rust & database newbie to working engine.

Why This Project?
Not just another key-value store, but a deep dive into real DB internals:

• Segmented logs: write amplification, log-structured storage
• In-memory indexing: speed/memory tradeoffs
• Compaction: space reclamation, speed vs durability
• Bloom filters: fast "not-found" lookups
• Index snapshots: instant restarts
• HTTP API: async & production-tested

Core Engine

• Durable, append-only log (fsync guarantees)
• Automatic segmented architecture
• Lightning-fast reads (O(1) in-memory HashMap index)
• Background compaction (auto space reclaim)
• CRC32 checksums for data integrity
• Index snapshots (5ms restarts)
• Tombstone deletions
• Bloom filters (negative lookups)

Production Ready

• HTTP REST API (Axum)
• Interactive CLI (REPL)
• /metrics and /health endpoints
• Docker & docker-compose support
• Criterion & k6 benchmarks
• Full CI/CD (build, lint, test)
• Rate limiting (100MB body max)
• Unit/integration test suite

Developer Experience

• Rich docs, many examples
• Modular, clean codebase
• Easy config via env vars
• Makefile with all tasks
• Pure safe Rust (no unsafe)

Prerequisites:

• Rust 1.75+ (install)
• Git

git clone https://github.com/whispem/mini-kvstore-v2
cd mini-kvstore-v2
cargo build --release
cargo test --release

REPL CLI:

cargo run --release
# mini-kvstore-v2 (type help for instructions)
# > set key "value"
# > get key
# > list
# > compact
# > quit

HTTP server:

cargo run --release --bin volume-server
# (config: PORT, VOLUME_ID, DATA_DIR env vars)

• /health: health/status
• /metrics: server stats
• POST /blobs/:key: store blob
• GET /blobs/:key: fetch blob
• DELETE /blobs/:key: delete blob
• GET /blobs: list all keys

Example:

curl -X POST http://localhost:8000/blobs/user:123 -d "Hello, World!"
curl http://localhost:8000/blobs/user:123
curl -X DELETE http://localhost:8000/blobs/user:123

[CLI / HTTP Client]
        │
    [Axum Server]
        │
   [Blob Storage]
        │
   [KVStore Core]
      /  |  \
 [Index][Segments][Bloom]

• Append-only data segments on disk
• HashMap and Bloom filter in RAM for O(1) queries
• Snapshot/compaction for instant restart & small disk use

Apple M4, 16GB RAM:

• Writes: ~240,000 ops/sec
• Reads: ~11M ops/sec (in-memory)
• Compaction: ~80,000 keys/sec

Run:

cargo bench
./run_benchmark.sh

Standalone:

docker build -t mini-kvstore-v2:latest .
docker run -d -p 8000:8000 -v $(pwd)/data:/data --name kvstore mini-kvstore-v2:latest

Cluster:

docker-compose up -d
# nodes: localhost:8001 ... :8003
docker-compose logs -f

cargo test --release
cargo test --release --test store_integration
make pre-commit    # fmt, clippy, test

• Extensive unit/integration + HTTP

• 80% coverage (goal)

• Benchmarks: Criterion + k6

Click to expand

• Append-only log, crash recovery, compaction
• Bloom filters, index snapshot
• REPL, HTTP, Docker, CI/CD, metrics
• Range queries, WAL, compression
• Replication, LSM, Prometheus, Admin UI

• Append-only: max I/O, easy recovery
• In-memory index: trade memory for O(1)
• Bloom filter: instant "not found"
• Rust: safety, perf, reliability

• Database Internals
• DDIA
• sled
• Bitcask Paper

• 🐛 Report bugs
• 💡 Suggest features
• 🧪 Add tests
• 📖 Improve docs
• ⚡ Performance PRs
• How to contribute

• Discord: Rust Aix-Marseille
• LinkedIn: Rust Aix-Marseille
• GitHub Discussions, Issues

MIT — see LICENSE

Em' (@whispem)

From languages to Rust/DB internals in 3 weeks. See JOURNEY.md.

"The best way to learn is to build."

Built with ❤️ in Rust • Back to Top