yasdb

yasdb is a Durable Streams server. It runs as a plain net/http server. It stores data with SlateDB, an LSM engine that uses only object storage for durability.

See docs/DEPLOY.md to deploy yasdb on Fly.io with Tigris.

• Go 1.25 or later
• A C toolchain (cgo). The SlateDB binding links a native library through cgo.
• lib/libslatedb_uniffi.so (checked in for linux/amd64)

The Makefile sets the cgo linker and loader flags for the bundled library:

make build
make run ARGS="-addr :4437 -data ./yasdb-data"

Day to day, you mainly choose where data lives: -store / -data / -db (see Storage backends below). Notifier durability, coalesced live-reader wakeups, and the in-memory record cache default to the settings this project's benchmarking found fastest — no flag needed to turn them on. The SlateDB compactor knobs stay off by default (the engine's own defaults apply). See Tuning below for when to use either group of settings.

Live-reader wake coalescing (a 5ms window) and the in-memory record cache are always on. See BENCHMARKS.md's "Live fan-out" section for the measurements behind that choice. There is no flag to disable either: correctness is the same either way, and the throughput gain under real fan-out is large (up to 5× at 500 concurrent readers on one stream).

The compactor knobs are still an open question; they are not a settled default. This project's local tuning sweep (BENCHMARKS.md's "Compaction stall under load") ran the load generator and the server on one shared box. It found no consistent win, and the result cannot be trusted either way: everything on the box was competing for the same cores. These knobs need re-testing with the load generator and the server on separate Fly machines before you can draw a real conclusion. See docs/DEPLOY.md's "Reading metrics from a load test" section to pull the numbers back afterwards.

Local file store (default). Leave -store empty. Data lands under -data. yasdb creates the directory if needed, and the data survives restarts.

make run ARGS="-data /var/lib/yasdb -addr :4437"

Object store (S3 / R2 / Tigris). Pass the bucket as -store and put the prefix in -db. The store URL must be a bare bucket; yasdb rejects a URL that has a path. Credentials and the endpoint come from the standard AWS environment variables (object_store conventions).

export AWS_ACCESS_KEY_ID=…  AWS_SECRET_ACCESS_KEY=…  AWS_REGION=auto
export AWS_ENDPOINT=https://fly.storage.tigris.dev      # S3-compatible endpoint
make run ARGS="-store s3://my-bucket -db yasdb -addr :4437"

In-memory (dev only; yasdb persists nothing):

make run ARGS="-store memory:/// -addr :4437"

Only one process may write to a given store/prefix at a time. SlateDB is single-writer per database. See docs/DEPLOY.md for a full Fly.io + Tigris deployment.

-flush sets the durability interval. An append blocks until the next WAL flush, so -flush is both the single-append latency floor and the rate at which yasdb writes WAL objects (this drives request cost on real object storage). -durability decides whether concurrent appends pipeline. Pick your flags from the shape of your workload:

BENCHMARKS.md has the numbers behind these knobs, and behind the always-on live-reader wake coalescing and record cache.

-metrics-addr :9091 (or YASDB_METRICS_ADDR=:9091) starts a Prometheus /metrics endpoint on its own address. It serves every metric the SlateDB engine registers natively: write/WAL throughput, L0 SST count, stall and backpressure counters, compaction throughput and job counts, object-store request rate/errors/latency (as histograms), block-cache hit rate, bloom-filter false-positive rate, and GC/expiry counts. It is off by default and kept off the main -addr. On Fly, fly.toml's [metrics] block points at this port; Fly scrapes it over the private network (not the public one) and surfaces it automatically in the org's built-in Grafana at fly-metrics.net. See deploy/grafana-slatedb-dashboard.json for a starter dashboard — import it, or drop it in Grafana's dashboard provisioning path.

make run ARGS="-metrics-addr :9091"
curl localhost:9091/metrics

GET /__admin/streams lists every stream on the server: content type, closed status, record count, fork status, and live reader count split by long-poll vs SSE. The list is cursor-paginated (?cursor=/?limit=, default page size 50, capped at 500):

curl "localhost:4437/__admin/streams?limit=50"
# {"streams":[{"path":"/demo/chat","contentType":"application/json","closed":false,
#              "records":42,"isFork":false,"readers":{"longPoll":0,"sse":1}}],
#  "nextCursor":"..."}

For a stream with recent activity, reader counts and record counts reflect the resident streamer's live in-memory state. For a dormant stream, they fall back to durable state; a dormant stream by definition has no parked readers. This endpoint adds no new read path for stream contents: offset-based browsing reuses the existing catch-up GET <path>?offset= endpoint and its Stream-Next-Offset cursor.

internal/state implements the Durable Streams State Protocol. This is a JSON message convention (type/key/value/headers.operation) layered on ordinary application/json streams. It gives change events (insert/update/delete) and control events (snapshot boundaries, reset) enough meaning to materialize into queryable entity state. It needs no changes to the base stream engine; see DESIGN.md for why.

GET /__state/<stream-path> is a server-side convenience built on that protocol: it replays a stream's full history through a state.Materializer and returns the current materialized state, grouped by entity type, plus a streamNextOffset to resume live-tailing from. Use it from a non-JS client, or anywhere you want to curl the current state without replaying raw history yourself.

curl localhost:4437/__state/demo/chat
# {"generatedAt":"...","streamNextOffset":"...","collections":{"user":[...],"message":[...]}}

# create a JSON stream
curl -X PUT localhost:4437/orders -H 'Content-Type: application/json'

# append a batch (flattened into two messages) and one more
curl -X POST localhost:4437/orders -H 'Content-Type: application/json' -d '[{"id":1},{"id":2}]'
curl -X POST localhost:4437/orders -H 'Content-Type: application/json' -d '{"id":3}'

# read from the start
curl 'localhost:4437/orders?offset=-1'      # -> [{"id":1},{"id":2},{"id":3}]

# live tail via SSE
curl -N 'localhost:4437/orders?offset=now&live=sse'

# close (EOF) and delete
curl -X POST   localhost:4437/orders -H 'Stream-Closed: true'
curl -X DELETE localhost:4437/orders

By default, the suite runs against an in-memory fake store, so it needs no native library to execute. Set YASDB_TEST_BACKEND=slatedb to run it against the real SlateDB store instead.

make test ARGS="-race"                                   # full suite, sync durability, fake store
YASDB_TEST_DURABILITY=notifier make test ARGS="-race"    # notifier durability
YASDB_TEST_BACKEND=slatedb make test ARGS="-race"        # against the real SlateDB store
make bench                                               # throughput / allocation benchmarks (real store)
make fuzz                                                # property: cache bytes == store bytes (Ctrl-C to stop)
MAELSTROM=/path/to/maelstrom/maelstrom make maelstrom DUR=notifier        # Jepsen log-consistency check
MAELSTROM=/path/to/maelstrom/maelstrom make maelstrom DUR=notifier READ=long-poll  # …over the live-read + cache path
MAELSTROM=/path/to/maelstrom/maelstrom make maelstrom-chaos DUR=notifier  # crash-injection durability check