Traza

A trace datastore with first-class LLM and agent observability — one binary from laptop to cluster.

Traza (Spanish for "trace") ingests OpenTelemetry or plain-JSON spans over HTTP, stores them durably, and answers trace lookups, filtered searches, and token/cost analytics in milliseconds — with a trace browser and no infrastructure to stand up. Two dependencies (serde, serde_json), no external database, and one deployment story at every size: today a single node that starts in milliseconds; the designed trajectory is replicated, highly available clusters of the same binary.

Grab a release — the server binary with the dashboard already built, no Rust or Node needed. macos-aarch64 below; linux-x86_64 and linux-aarch64 archives are named likewise (musl-static, so any distribution works):

curl -LO https://github.com/toshish/traza/releases/download/v0.22.1/traza-0.22.1-macos-aarch64.tar.gz
tar xzf traza-0.22.1-macos-aarch64.tar.gz && cd traza-0.22.1-macos-aarch64
./traza-server --data-dir ./data --port 8080
# open http://localhost:8080

The macOS binary is not yet notarized: a browser download gets quarantined by Gatekeeper (curl does not set the flag; xattr -d com.apple.quarantine traza-server clears it if your download method did).

Or the container — FROM scratch, nothing in it but the binary and the dashboard, running as uid 65534. The non-loopback bind refuses to start without a token, so mint one you keep:

TOKEN="rw:$(openssl rand -hex 16)"
echo "$TOKEN"    # the dashboard and API will ask for this
docker run -p 8080:8080 -v traza-data:/data \
  -e TRAZA_TOKENS="$TOKEN" ghcr.io/toshish/traza:v0.22.1

Or from crates.io — cargo add traza for the library, cargo install traza --locked --bin traza-server for the server — or the full tree with the dashboard:

cargo build --release
(cd ui && npm ci && npm run build)          # builds the dashboard into ui/dist
./target/release/traza-server --data-dir ./data --port 8080
# open http://localhost:8080 — the server serves ui/dist

Release archives carry SHA256SUMS, GitHub build-provenance attestations (gh attestation verify traza-*.tar.gz --repo toshish/traza), and third-party license material in THIRD_PARTY_NOTICES.md.

curl -X POST http://localhost:8080/v1/spans -H 'Content-Type: application/json' \
  -d '[{"trace_id":"trace-1","span_id":"span-1","name":"charge","service":"checkout",
        "start_time_unix_nano":1700000000000000000,
        "end_time_unix_nano":1700000000002500000,
        "status":"ok","attributes":{"region":"us-east"}}]'
# {"accepted":1,"durability":"wal"}

curl http://localhost:8080/v1/traces/trace-1

Full walkthrough: Getting started.

The complete set lives in docs/, organised by what you are doing.

Using Traza — getting started · data model · ingest · HTTP API reference · LLM semantics · MCP server · trace browser

Operating Traza — deployment · durability · administration · monitoring · capacity · configuration reference

Changing Traza — architecture · invariants · module map · testing · benchmarking · segment format · CONTRIBUTING.md

Direction — roadmap · HA design

Most tracing backends make you assemble a fleet before the first span: a column store or search cluster, a queue in front, an operator keeping it healthy. Traza's bet is that a trace datastore should scale like a database, not like a pipeline — one binary whose deployment grows with you instead of a different architecture at every size:

• Start on one machine in seconds. A single process stores everything under --data-dir and needs nothing else — right-sized for a laptop, a CI job, a single-host service, an edge box, or the AI agent you're debugging right now.
• Scale by adding nodes, not systems. The engine's foundations — immutable segments, idempotent primary-key ingest, journaled compaction — were chosen to replicate. The HA design (quorum-replicated logical log, validated full-state snapshots for catch-up) is the committed trajectory. Today's scope is single-node; see Status.
• Built for LLM and agent workloads. Sessions, token and cost analytics, prompt/completion capture with large-payload offloading, post-hoc evals and feedback, and one-command dataset export — first-class, not bolted on.
• OpenTelemetry-compatible, OpenLLMetry-native. Point any OTel SDK at it with two environment variables. Traza follows the OpenLLMetry standard (gen_ai.* / traceloop.*), so instrumented apps get sessions and token/cost analytics with no attribute renaming.
• Small enough to trust. Two direct dependencies; HTTP, threading, and file I/O are the Rust standard library. #![forbid(unsafe_code)]. The budget is paid for in the open: the SHA-256 behind content-addressed payloads is implemented in-crate against the FIPS 180-4 test vectors and is never used for authentication or tamper-proofing, and the 128-bit index hash is not a cryptographic commitment — every index probe is re-verified against the stored record. Every performance number is measured by a bundled benchmark, with anything extrapolated marked as such.
• Crash-safe by construction. Immutable segments written by write-temp, fsync, atomic rename; recovery loads only complete segments and heals crash artifacts.

Span identity is a primary key. (trace_id, span_id) uniquely names a span; re-ingesting it replaces the stored version (last write wins). Client retries are idempotent and never create duplicates — no idempotency key, no client-side deduplication. See the data model.

Two ingest surfaces. POST /v1/spans takes a plain JSON array; POST /v1/traces takes OTLP/HTTP as binary protobuf or JSON. Point any OTel SDK at Traza with OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf. gRPC is not served. See ingest.

LLM and agent telemetry lands queryable without translation. gen_ai.*, llm.usage.*, and traceloop.* attributes drive sessions, provider/model rollups, and token/cost analytics directly:

curl 'http://localhost:8080/v1/sessions?limit=20'          # per session: spans, traces, tokens, cost, errors
curl 'http://localhost:8080/v1/sessions/chat-4711'         # + the per-trace breakdown
curl 'http://localhost:8080/v1/stats/llm?group_by=model'   # or provider | service | session | day

Prompt and completion payloads above --payload-threshold-bytes are offloaded to a content-addressed store and replaced inline by a $payload reference, so a repeated system prompt is stored once. Evals and human feedback attach after the fact via POST /v1/annotations without mutating spans, and GET /v1/export turns any search into a streaming NDJSON dataset. Conventions and query recipes: LLM semantics.

Your agent can read its own traces. --mcp serves a Model Context Protocol endpoint from the same binary — ten tools shaped like the questions people arrive with (what is failing, what is slow, where did the money go) rather than one per HTTP route. Results are bounded in tokens instead of rows: twenty spans by default, prompts omitted until asked for, every result capped, and every truncation stated, because a silently shortened answer gets reported as a complete one. Stored span text is returned inside a block marked untrusted and never reaches a tool description or an error message — and the server holds no fetcher, shell or outbound path for an injected instruction to actuate. Off by default; claude mcp add --transport http traza http://localhost:8080/v1/mcp once it is on. Run examples/mcp-demo/run.sh to watch an agent work through a seeded store without wiring anything up.

A trace browser, served from its build output. The dashboard is a React app in ui/ that traza-server serves straight from ui/dist — nothing is compiled into the binary, so building the server needs no Node toolchain and a rebuilt UI is picked up without a restart. A packaged binary without a build ships the API only: / then returns a 404 explaining how to build it, and startup logs every path searched. See trace browser and deployment.

Durability you choose and the server states. --durability is buffered, wal (default), or flushed, and every ingest response echoes the mode so a client never has to guess what its 200 meant. One caveat stated plainly: fsync on macOS does not flush the drive's own write cache, so a power cut there can still lose an acknowledged write; a kill -9, a panic, or an OS crash cannot, on either platform. See durability.

Memory scales with how many distinct things you index, not how big they are. Segments are file-backed and payloads are read on demand, so a store larger than RAM serves correctly. The part that used to break on LLM traffic was the index itself: through segment format v3 it was keyed on the whole attribute value, so indexed prompt text stayed in RAM verbatim at RSS ≈ 1.44 × the prompt text ingested — O(data), not O(indexes). v4 keys it on a 128-bit digest instead, and the same corpus measures 391 MiB → 21.6 MiB on 256 MiB of all-distinct 2 KiB prompts. Enum-valued attributes are unchanged and were always cheap: 10M spans with six of them open in 846 MiB, at 8 bytes per span per indexed attribute. Filtered search costs one index probe per segment, which is what size-tiered compaction (on by default) exists to bound. Measured figures and their trade-offs: capacity.

Content search that doesn't put the text back in RAM. ?content=refund finds spans by the words in their prompts, completions, tool arguments and events. Segments carry a Bloom filter over the words in each 128-record block, stored bit-sliced so a probe reads tens of bytes per segment rather than the whole filter. Measured on 200,000 spans holding 145 MiB of text: a selective term returns in 1.5 ms against 1,258 ms scanning, for +0.1% on disk and ~2 KiB resident per segment. It is word matching, not substring matching — a word index cannot soundly drive a substring query — and when nearly every span matches it correctly buys nothing. A value large enough to be offloaded to the payload store is searchable only within its inline preview, which at the 256 KiB default threshold is almost nothing. See content search.

Every parameter, response shape, and error is in the HTTP API reference.

The engine embeds in your own process — same durability, no server:

use traza::{Config, SpanFilter, Store};

let store = Store::open("./data", Config::default())?;
store.ingest(span)?;          // buffered; flushes automatically at the threshold
store.flush()?;               // or force a durable segment now
let slow = store.query(&SpanFilter {
    service: Some("checkout".into()),
    min_duration_ns: Some(5_000_000),
    ..SpanFilter::default()
})?;

A complete, runnable version is in ingest. A data directory has exactly one writer, so an embedding process must not also run a server against it.

Two layers with one contract: never lose a completed write, never serve a torn one.

The storage engine buffers spans in memory, appends them to a write-ahead log, and flushes sorted, immutable segment files — JSON payloads with embedded record-offset, trace, and attribute indexes — via write-temp, fsync, atomic rename. Opening a store parses only the indexes; spans materialize on demand. Filters narrow candidates through the indexes, then re-verify every predicate against the parsed span: an index accelerates a filter, it never changes its semantics.

The HTTP server is a deliberately small HTTP/1.1 implementation on std::net, bounded by concurrent connections rather than a queue, in front of the engine — which is its only datastore. There is no server-side log or side index; restart durability is the engine's.

Deeper: architecture · invariants · segment format.

Measured on macOS/aarch64 (10 hardware threads) by the bundled benchmarks over corpora ingested through the real HTTP path:

• Sustained ingest: 208,973 spans/s at 16 concurrent clients in wal mode, measured on an idle machine (ingest.md); the single-client canonical record predates the 0.20 compaction rework and is queued for re-measurement
• Trace lookup: p95 0.64 ms · Attribute-filtered search: p95 3.3 ms (1M-span corpus)
• Compaction is worth 16–28x on filtered search at 100M spans, and the segment-size cap is worth another 3–4x on top — at a real cost in memory and ingest throughput

Full percentiles, the 10M and 100M scaling runs, the ingest matrix, and an honest list of what is not measured: capacity. The underlying records are canonical-corpus.md and ingest.md, both rewritten by the benchmarks themselves — run them on your hardware rather than trusting ours.

Where Traza is expensive: disk. Segments are uncompressed JSON plus indexes, so an ordinary span corpus costs 1.8–2.1x the bytes a client sent — worse than Elasticsearch, far worse than a Parquet-and-object-storage system. The exception is the workload the payload store was built for: 3.1 GiB of pinned agent context measures 121:1 in Traza's favour, because a byte-identical context above the offload threshold is stored once. Measured both ways, next to OpenObserve's published Elasticsearch comparison, in storage comparison and storage.md.

Traza is pre-1.0 and honest about it: on-disk formats may change between 0.x versions, and single-node is the current scope. Shipped and load-bearing today: durable segment storage with a write-ahead log and crash recovery, size-tiered compaction, OTLP protobuf/JSON ingest, sessions and cost analytics, payload offloading, annotations, streaming export, bearer auth, Prometheus metrics, and the ui/ trace browser served from its build output.

Known architectural gap: query-visible state lives in several independent recovery domains (the write-ahead log and buffer, segments, annotations, payload files), and nothing yet names one state they all agree on. Backup, export, retention and replication are consequently four mechanisms rather than one. The generation/checkpoint boundary that fixes the class is designed in docs/generations-design.md and scheduled before 1.0.

The destination is bigger than one node. The full product roadmap — durable v1 foundations, then replicated HA clusters and agent-native debugging depth, then columnar analytics at billion-span scale, then the enterprise control plane — lives in docs/roadmap.md, with the HA architecture detailed in docs/ha-design.md. Same binary, same API, at every phase.

Deliberately out of scope: a metrics/logs suite, embedded eval models, general SQL, and framework SDKs — the roadmap explains why.

See CONTRIBUTING.md. The short version: stable Rust is the only dependency, ./ci.sh is the merge bar, and new dependencies need a reason.

Copyright © 2026 Toshish Jawale. Licensed under the Apache License, Version 2.0. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be licensed as above, without any additional terms or conditions.