Stop provisioning costly Kafka clusters or Redis brokers just to buffer webhooks and audit trails. walspool v1.0 combines a 1,132,000 ops/s sequential disk Write-Ahead Log (128KB Group Commit + IEEE CRC32) with a thread-safe in-memory Ring Buffer broadcasting Server-Sent Events (SSE) at > 1,000,000 req/s and < 15µs trace queries.
Two specialized engines work in tandem behind pure Black-Box boundaries: durable append-only disk logging and an ultra-fast in-memory streaming hub.
Engineered for zero data loss. Direct binary serialization with an amortized userspace buffer prevents random disk seek penalties while guaranteeing durability across abrupt power cuts and container evictions.
.tmp → checkpoint.meta) with directory fsync.A high-speed in-memory indexing buffer that eliminates external logging infrastructure for live telemetry, distributed tracing, and real-time debugging streams.
trace_id and service in under 15 microseconds.WithObserver(hub)
Live Broadcast
Measured on Intel Core i7-1255U (NVMe SSD, Linux 6.8) with Go benchmark suite
| Benchmark Function | Throughput | Latency | Memory / Op | Allocs / Op |
|---|---|---|---|---|
| BenchmarkHub_Ingest | 2,360,859 ops/s | 492.7 ns/op | 203 B/op | 1 alloc/op |
| BenchmarkFileStorage_Append_SyncInterval_128KB | 1,132,000 ops/s | 695.7 ns/op | 238 B/op | 1 alloc/op |
| BenchmarkHub_QueryByTraceID | 718,039 ops/s | 1,541 ns/op (1.54 µs) | 1,440 B/op | 11 allocs/op |
| BenchmarkSpoolerEnqueue_InMemory | 1,028,797 ops/s | 1,130 ns/op (1.13 µs) | 1,148 B/op | 1 alloc/op |
All endpoints exposed by the cmd/sidecar process for polyglot systems integration.
| Method | Endpoint Path | Functionality & Contract | Latency | Status |
|---|---|---|---|---|
| POST | /v1/logs | OpenTelemetry (OTLP) Ingestion: Native OTLP/HTTP receiver (Protobuf & JSON). Auto-extracts service name, trace ID, and severity into NVMe WAL and MemoryLogHub. | < 20 µs | 200 OK |
| POST | /v1/enqueue | Dual Ingestion: Atomic append to disk WAL with CRC32 + instant MemoryLogHub indexing. Triggers SSE broadcast. | < 15 µs | 202 Accepted |
| GET | /v1/logs |
Historical Query: Retrieves logs from Ring Buffer filtered by trace_id, service, and level.
|
< 15 µs | 200 OK |
| GET | /v1/logs/stream | Real-Time SSE: Non-blocking event streaming via Server-Sent Events. Includes 10s keepalive heartbeats. | < 1 ms | 200 (SSE) |
| GET | /v1/logs/stats |
Hub Observability: Returns capacity, current size, total ingested, active streams, and dropped_events.
|
< 10 µs | 200 OK |
| GET | /metrics | Prometheus / OpenMetrics: Ingestion rate, WAL buffer bytes, queue backlog, and retry error counters. | < 20 µs | 200 OK |
| GET | /healthz & /readyz | Kubernetes Probes: Liveness and readiness health checks confirming daemon and storage health. | < 5 µs | 200 OK |
| POST | /flush | Synchronous Drain: Flushes all in-flight committed records to downstream HTTP/Kafka/S3 sink. | I/O Bound | 200 OK |
Deploy Walspool seamlessly into your existing containerized environments. Visual schemas, Docker Compose snippets, and zero-code migration patterns.
Every application container (Node.js, Python, PHP, Ruby) has its own local Walspool companion running in the same network namespace.
services:
api-service:
image: my-app:latest
network_mode: "service:walspool"
environment:
- WALSPOOL_URL=http://127.0.0.1:9099/v1/enqueue
depends_on:
- walspool
walspool:
image: ghcr.io/yohannhommet/walspool:v1.0.0
environment:
- WALSPOOL_ADDR=:9099
- WALSPOOL_DATA_DIR=/data/spool
- WALSPOOL_SINK_URL=https://sink.example.com/v1/events
- WALSPOOL_BATCH_SIZE=100
volumes:
- wal-data:/data/spool
volumes:
wal-data:
Consolidate event spooling on a single VM or cluster node. Multiple microservices push logs to a shared Walspool container over the internal Docker network.
services:
walspool:
image: ghcr.io/yohannhommet/walspool:v1.0.0
ports:
- "9099:9099"
environment:
- WALSPOOL_SINK_URL=https://lake.example.com/v1/events
- WALSPOOL_MAX_RECORDS=100000
billing:
image: billing-app:v1
environment:
- WALSPOOL_URL=http://walspool:9099/v1/enqueue
auth:
image: auth-app:v2
environment:
- WALSPOOL_URL=http://walspool:9099/v1/enqueue
Stream events in real time to web browsers and dev consoles via Server-Sent Events (SSE) while persisting to disk with zero lock contention.
location /v1/logs/stream {
proxy_pass http://walspool:9099;
proxy_http_version 1.1;
proxy_buffering off;
proxy_cache off;
proxy_set_header Connection '';
add_header X-Accel-Buffering "no";
proxy_read_timeout 24h;
}
Designed for IoT gateways, retail POS terminals, or remote sites with intermittent or unreliable internet connectivity.
Full technical specifications available in docs/architecture/:
Detailed contracts, invariant guarantees & latency models
Walspool resolves configuration strictly: CLI Flags > Environment Variables > Hardcoded Defaults.
| CLI Flag | Environment Variable | Default Value | Description & Validation Rule |
|---|---|---|---|
| -addr | WALSPOOL_ADDR | :9099 | HTTP daemon TCP bind address. Must not be empty. |
| -data-dir | WALSPOOL_DATA_DIR | ./data/spool | Directory path on local disk for append-only .wal and checkpoint.meta. |
| -sink-url | WALSPOOL_SINK_URL | "" (stdout console) | Target HTTP URL to deliver batched events to. If omitted, logs to stdout. |
| -batch-size | WALSPOOL_BATCH_SIZE | 50 | Number of records per batch drain to the Sink. Must be > 0. |
| -flush-ms | WALSPOOL_FLUSH_MS | 50 | Maximum interval in milliseconds before flushing partial batches. Must be > 0. |
| -max-records | WALSPOOL_MAX_RECORDS | 50000 | Storage capacity quota before triggering backpressure rejection (503 spool_full). |
| -hub-capacity | WALSPOOL_HUB_CAPACITY | 50000 | Fixed circular Ring Buffer size in memory for real-time observability. Must be > 0. |
// 1. Import official Go module
import "github.com/YohannHommet/walspool"
// 2. Initialize Disk Storage (Engine 1 - Corten Rust) and In-Memory Hub (Engine 2 - Acid Lime)
storage, _ := walspool.NewFileStorageEngine("./data/spool", 50000)
hub := walspool.NewMemoryLogHub(50000)
// 3. Wire Dual-Engine via WithObserver
cfg := walspool.DefaultConfig()
spool, _ := walspool.New(cfg, storage, sink, nil, walspool.WithObserver(hub))
defer spool.Close()
// 4. Ultra-fast append (< 1 µs) - automatically persisted and indexed
_ = spool.Enqueue(ctx, "orders.checkout", []byte(`{"trace_id":"tr-9941","total":149.00}`))
// 5. Query historical logs in < 15µs
logs := hub.Query(walspool.LogQuery{TraceID: "tr-9941", Limit: 10})
| Capability | Direct HTTP | Redis / Celery | Kafka / AWS SQS | walspool v1.0 |
|---|---|---|---|---|
| Ingestion Latency | 50–800 ms (Stalls API) | 2–5 ms (Network roundtrip) | 5–15 ms (Network roundtrip) | 0.69 µs (Direct Disk Log) |
| Crash Durability (SIGKILL / OOM) | ❌ 0% (Complete Loss) | ⚠️ Loses un-fsync'd RAM | ✅ High (Distributed cluster) | ✅ 100% (WAL + CRC32) |
| Real-Time Observability | ❌ None | ⚠️ Pub/Sub (Drops offline) | Requires consumer workers | ✅ Native SSE (> 1M req/s) |
| Historical Trace Lookup | N/A | Scan keys (Slow) | Requires Elasticsearch / ClickHouse | ✅ < 15 µs In-Memory Index |
| Infrastructure Cost | $0 | $200–$600 / mo | $800–$3,500 / mo | $0 (In-process) |
| Maintenance Overhead | Zero | Connection pools, leaks | Partitions, brokers, rebalancing | Zero (Single Go Import) |
Calculate your annual savings by replacing distributed brokers with walspool.
Walspool utilizes a 128KB userspace write buffer with SyncInterval policy (periodic 50ms fsync). Sequential appends amortize kernel syscalls and drive PCIe NVMe throughput to saturation without per-write seek stalls.
No. The LogHub is implemented as a strict, fixed-size circular Ring Buffer (default 50,000 entries). When full, new writes overwrite the oldest slots in O(1) time. Secondary indices (byTraceID and byService) automatically prune stale references, guaranteeing zero memory leaks.
Walspool reads checkpoint.meta on boot and scans forward in the WAL log. Every record is verified with IEEE CRC32. If an abrupt crash resulted in a partial or torn write, Walspool truncates the log file exactly to the last verified record boundary and resumes zero-loss dispatch.
Zero CGO. Walspool is 100% pure standard library Go. It compiles with CGO_ENABLED=0 into a tiny, static binary compatible with distroless/scratch containers on Linux, macOS, Windows, and ARM architectures.
Schedule a 15-minute technical review with our systems architects. We will inspect your event pipelines, verify crash scenarios, and build your custom integration blueprint.