⚡ Rust Low-Latency Trading Lab

A deterministic electronic trading core in Rust, with a measured latency pipeline, a TypeScript control plane, and digital-asset settlement on Solana and Canton.

This is an open-source engineering lab. It exists to show how a low-latency trading system is actually built: integer-only domain types, a hand-specified binary market-data protocol, a price-time limit order book, deterministic matching, pre-trade risk, bounded single-writer concurrency, byte-exact replay, honest measurement, and a settlement boundary that stays off the execution path.

It is not a production exchange, broker, or high-frequency trading platform, it contains no trading strategy, and it is not evidence of production HFT deployment. Every performance number below comes from a real run on the machine named next to it.

• Architecture
• What the engine guarantees
• Quick start
• Deterministic replay
• Pre-trade risk
• Benchmarks
• Control plane
• Settlement
• Testing
• Repository layout
• Development workflow
• License

The hot path is a single writer. Market data is decoded, normalized, and handed to one engine thread that owns all trading state; risk, book, and matching run inline on that thread; outputs leave over a bounded queue. Everything operational — HTTP, JSON, SQLite, metrics, blockchains — lives on the cold path and can never block execution.

flowchart TD
    MD[Market data feed] --> DEC[Binary feed decoder]
    DEC --> Q1[[Bounded SPSC queue]]
    Q1 --> ENG

    subgraph ENG[Rust engine thread · single writer]
        RISK[Pre-trade risk] --> BOOK[Limit order book]
        BOOK --> MATCH[Matching]
    end

    ENG --> Q2[[Bounded SPSC queue]]
    Q2 --> OUT[Trades & execution reports]
    OUT --> JRN[CRC-framed journal & replay]

    JRN --> GW[Rust control gateway]
    GW --> API[TypeScript control plane]
    API --> DB[(SQLite projection & outbox)]
    DB --> SOL[🔷 Solana settlement]
    DB --> CAN[🔐 Canton settlement]

    API -. cold path, never blocks the engine .-> GW
    GW -. bounded control commands .-> ENG

Nothing on the dashed edges can stall matching: control commands are applied between inputs from a bounded queue, and the kill switch is a single atomic latch the engine reads before each input.

Order semantics. Limit orders are good-til-cancelled and rest; market orders are bounded by a protection price derived from the reference market and never rest. Self-trading is allowed and reported normally, because this is a lab, not a venue with self-match prevention rules.

Replace semantics. A same-price quantity decrease keeps queue priority. A price change or a quantity increase loses priority. The new total may never fall below the cumulative filled quantity. A replace may not change the side or the order type, and the resting order is the canonical source of both. Every check runs before any mutation, so a rejected replace leaves the order untouched.

Market data. Feed prices are validated against the instrument price domain. A snapshot level, level update, or trade outside that domain puts the feed into a gap rather than becoming a reference price, so a hostile or corrupt feed cannot move the collar. The retained view is a configured top-N ladder per side (default 256), not a full market book: best bid and ask stay correct, deeper valid levels are dropped, and snapshots are truncated to that depth.

Requires a stable Rust toolchain (1.85+). Everything below runs offline.

# Build and run the whole test suite
cargo test --workspace

# Generate a deterministic binary feed, then replay it
cargo run --release -p simulator -- generate --seed 1 --events 500 --out /tmp/lab.feed
cargo run --release -p simulator -- replay --input /tmp/lab.feed

# Prove the run is reproducible: two replays, three matching digests
cargo run --release -p simulator -- verify --input /tmp/lab.feed

# Measure queue and pipeline latency at a stated offered load
cargo run --release -p simulator -- bench --wait busy-spin --rate 100000

replay prints the input, output, and state digests along with queue high-water marks; step walks the run one input at a time for debugging; bench prints the environment it measured on.

To run the operational stack, start the gateway and then the control API:

RLTL_CONTROL_TOKEN=dev-token cargo run --release -p control-gateway -- \
  --socket /tmp/rltl-control.sock --journal /tmp/rltl-engine.journal

cd apps/control-api && npm ci
RLTL_CONTROL_TOKEN=dev-token RLTL_API_TOKEN=dev-api npm start
curl -s localhost:8080/book/LAB-USD | jq

The engine is a pure state machine: apply(input, logical_time, output_buffer). It performs no I/O, reads no clock, and spawns no tasks, so the same inputs always produce the same outputs and the same final state.

There is one input stream. Market data, order requests, and control commands are sequenced at the point the engine applies them, not by the producers that supply them, so an operator action taken mid-run lands in the recorded stream in the order the engine actually saw it. Replaying that stream reproduces a controlled run exactly; the wall-clock moment the operator pressed the button is not reproduced, and does not need to be.

Three digests are recorded over canonical encodings, never over memory layout, pointers, hash-map order, padding, or wall-clock time:

• input digest — every input the engine applied, control commands included
• output digest — every trade, execution report, and state event it produced
• state digest — every piece of state that can change a future decision: sequences, next identifiers, the kill latch, feed state and epoch, the last trade price, the levels of a snapshot still in progress, visible market levels, book levels with their FIFO order, and per-account positions, reservations, limits, and dedup entries

Each variable-length section of the state encoding carries a tag and an entry count, so two different collection shapes cannot produce the same bytes. The encoding is versioned (STATE_SCHEMA_VERSION); the golden fixtures are regenerated whenever it changes.

Periodic checkpoints make divergence cheap to locate: instead of "the run differs", you get the first checkpoint where the digests split. Threaded runs are verified against single-threaded replay in the test suite, including runs with an account disable, a risk-limit change, and kill switch transitions, so neither concurrency nor operator action can change the result.

Risk runs inline on the engine thread, in a fixed order, before anything touches the book:

1. Duplicate request detection against a bounded per-account ring with an O(1) request-id index. Replay walks the ring, never the index map.
2. Field, account, and instrument validation
3. Kill switch and per-account enablement
4. Client sequence monotonicity
5. Market synchronisation and staleness
6. Maximum order quantity
7. Price collar, protection price, and maximum order notional
8. Worst-case position and gross exposure, including working-order reservations
9. Book capacity

Reservations are the part that is easy to get wrong: a working order already consumes limit capacity, so exposure is evaluated against the worst case if every resting order filled. Reservations are released on fill, cancel, and replace, and a test asserts they always match the live order set.

The kill switch is fail-closed: it is an atomic latch the engine reads before every input, so it takes effect even if the control queue is saturated. The latch is the only source of kill state, so engaging, releasing, and engaging again all reach the engine, and each transition is recorded once.

Real measurements only. Every number below was produced by the commands shown, on the machine described. Re-run them yourself; they will differ on your hardware.

Environment: Apple M5 Max, 18 logical cores, macOS 25.6.0 (aarch64), rustc 1.97.1, release profile, commit 5617115 with this working tree applied. This is a development laptop, not an isolated measurement host.

Dedup lookup, same filled window, indexed map versus a linear scan of the ring (the previous implementation). Indexed stays flat; scan grows with the window.

The two engine benchmarks measure batches of 1,000 orders against a freshly prepared core, because a single long-lived core fills its 4,096-order capacity and starts rejecting. Each batch asserts that every order took the intended path, and the bench binary prints an acceptance check before Criterion measures anything.

Boundary: feed thread → bounded SPSC queue → engine → output thread. No journal, no snapshots, no control API, no settlement. Two start points are measured separately and never combined:

• scheduled-arrival-to-report starts at the instant the input was due on the offered schedule, so it includes any time the producer spent behind that schedule
• enqueue-to-report starts when the producer actually handed the input over, so it excludes producer scheduling debt

Each offered rate is run six times: one warm-up run that is not published, then five measured runs. The tables give the median across the five runs, with the lowest and highest run in brackets, so the run-to-run spread is visible. These are not confidence intervals. Every individual run is printed by the command above. Workload: 200,000 generated events, 252,024 measured order reports per run, busy-spin wait strategy, 4,096-entry queues.

scheduled-arrival-to-report

enqueue-to-report

The gap between the two tables is the producer falling behind its own schedule: at 500,000 msg/s the generator missed its slot 848 times (median; one run missed 2,611), which moves median p99 from 1.96 µs to 3.54 µs. One of the five 500,000 msg/s runs was a tail outlier; the brackets are the lowest and highest run, not a trimmed set. Unpaced has no schedule, so the two boundaries are identical there. Queue high-water is a sampled approximation (every 64 successful pushes) and can lag the true depth by up to 63 events.

The unpaced row is deliberately included: with an infinite offered load the queue stays full, so the measurement becomes queue residence, not coordination latency. That is why latency is only quoted at a stated offered load.

These are development benchmarks taken on a macOS laptop, not isolated production-host latency claims. There is no core pinning, no isolated CPU set, no interrupt steering, and other processes were running. Tail percentiles in particular should be read as "what this laptop did five times in a row", not as a platform guarantee. Timing-call overhead is measured separately (timer_read_pair_baseline in the same bench command) and is not subtracted from latency results.

Boundary: the shipped run() path — canonical input recording and digests, the CRC-framed journal, snapshots every 1,000 engine sequences, and the same queue and runtime code the pipeline uses. Solana, Canton, Node, and external settlement are excluded. This path carries no per-event wall-clock stamp, so only whole-run throughput is reported, not latency percentiles. Workload: 5,000 generated events producing 5,261 inputs and 7,107 journal records; five measured runs each.

Durable acknowledgement costs roughly four thousand times the throughput of group commit on this laptop's filesystem. That is the honest price of one fsync per event, and it is why the gateway uses group commit and states that it is a visibility bound rather than a durability one.

The wait strategy dominates end-to-end latency at low load: an engine thread that parked wakes tens of microseconds late. Busy spin buys latency with a core, which is the trade a real venue makes.

Two processes sit between an operator and the engine, and neither can slow it down.

Rust control gateway. Tokio serves a local Unix socket using length-prefixed JSON. After bind the socket is mode 0600 (and a gateway-created parent directory is 0700), so the mode does not depend on umask. Every message carries a schema version and a command id, frames are bounded before parsing, and each response reports the engine sequence it was answered at. All 64- and 128-bit values cross the boundary as decimal strings, because JavaScript numbers cannot represent them exactly. Reads come from the journal and from periodic engine snapshots. Mutations and sensitive reads (snapshot, outputs, replay status, journal reset) require a shared token compared in constant time and are refused outright when no token is configured. health stays unauthenticated because it carries no account or order state. The listener caps concurrent connections (default 32) and applies a per-read timeout (default 15s) so a half-open client cannot hold a task forever.

Replay is a job, not a blocking request: POST /replay/start returns a job id immediately, the work runs on a dedicated thread rather than a Tokio worker, one job may run at a time, and only the latest 32 job records are retained.

Journal visibility and durability are different things, and the code keeps them apart. The output thread writes CRC-framed records through a buffer, and JournalSync chooses when that buffer is pushed:

The gateway runs GroupCommit, because it tails the journal and must not be reading a buffer that may never be published. Engine acknowledgement — the engine applied the input and emitted output — happens earlier and is a separate fact from either of these. If a journal write fails, the engine stops rather than continuing without a record.

Snapshots are bounded by depth, not by book size. The engine publishes a read model at a configured interval, on the trading thread, so capture reads at most the top depth levels per side and depth * 8 orders per instrument. It iterates the ordered structures directly: the book is never cloned and then truncated. A test measures capture against a 100-order book and a 4,000-order book at the same depth and asserts the cost does not follow the live order count.

The tail reader is incremental: it holds a file positioned at the last committed offset, reads only bytes appended since the previous poll, holds an incomplete trailing record until the writer finishes it, and reports corruption at the offset where it occurred. Gateway requests that touch the filesystem run on a blocking Tokio task rather than on a reactor worker.

TypeScript control API. Fastify with strict TypeScript and JSON Schema validation on every request. It tails the gateway, applies the output stream to a SQLite projection, and serves:

GET /health is the only unauthenticated HTTP route. Book, orders, trades, positions, settlements, and metrics require a Bearer token compared in constant time. The WebSocket checks bufferedAmount before each send and closes the socket (1013) if it exceeds a configured threshold (default 1 MiB), so a slow client cannot accumulate an unbounded output buffer.

The SQLite projection stores the engine run_id on trades and settlement identity. A restart on the same run resumes; a different run latches run_mismatch and stops applying until an explicit resync. A missing output sequence latches degraded the same way: later events do not clear it. Neither state is presented as fully healthy on /health, metrics, or projected reads.

There is deliberately no order entry in the operational API. Operational tooling changes limits and stops trading; it does not send orders.

Observability follows the same rule: labels are bounded to symbols, feed states, settlement statuses, and the fixed reject-reason enum. Order, account, and trade identifiers are never used as labels, and no log formatting happens on the trading thread.

Settlement is a boundary, not a step in execution. An executed trade stays executed even if settlement is delayed, unknown, or retried.

The projection groups unsettled trades into batches in a SQLite outbox. Each batch gets a settlement identity derived from the engine run, its trade range, and a canonical manifest hash over its contents. Retries reuse both, so a timeout can never create a second economic identity. That manifest hash is the outbox's own record of the batch; each venue additionally binds its settlement to the economics it actually applies, in the way that venue can prove — see below.

pending ──▶ submitted ──▶ confirmed        (terminal)
                    └──▶ unknown ──▶ submitted | confirmed | failed
                    └──▶ unknown ──▶ needs_operator   (automatic retries exhausted)

unknown is not a failure and is never rewritten to failed. A settlement is only marked failed when the venue rejected the batch before any economic effect. unknown retries use bounded exponential backoff with bounded jitter (base 500 ms, cap 60 s, eight automatic attempts). After that the row becomes needs_operator and stays economically unresolved. The pending query only returns rows whose next_attempt_at is due, so one stuck settlement does not head-of-line block newer work.

A proven pre-submit rejection can be released and requeued by an operator. unknown, confirmed, and needs_operator cannot: those must not mint a second economic identity. A manifest conflict also goes to needs_operator.

A minimal Anchor program holds collateral in PDA-isolated accounts and applies batch settlements: initialize_exchange, open_collateral, deposit_collateral, withdraw_collateral, freeze_account, and settle_batch. It validates signers, PDA seeds, account ownership, the mint and token program, the exchange authority, and every amount, and uses checked arithmetic throughout. Before a batch mutates anything it checks each supplied collateral account: writable, unique by public key, owned by this exchange, and at its canonical PDA address. That closes the classic duplicate-mutable-account hole where the same account appears twice in one batch and one leg overwrites another.

The manifest hash is proved on chain, not trusted. There is one canonical binary encoding of a batch: a 16-byte domain tag RLTL-SETTLE-V1 , the 16-byte settlement id, the 32-byte exchange key, the leg count as u16 little-endian, then each leg in order as payer collateral key (32 bytes), payee collateral key (32 bytes), amount as u64 little-endian. Note that a leg is identified by the collateral account key, never by its index into the remaining accounts. The program resolves every leg, hashes that preimage with sha256, and rejects the batch with ManifestMismatch unless the result equals the supplied hash — before any balance moves. The TypeScript client builds the same bytes with node's built-in crypto, and a golden vector (two legs, 210-byte preimage, hash d11c5555…) is asserted from both Rust and TypeScript, so the two encoders cannot drift apart. Changing an amount, swapping a payer or payee, or reordering the legs while keeping the old hash is rejected, and each of those is a test.

Idempotency comes from a receipt PDA keyed by the settlement identity: submitting the same identity twice fails, and the client resolves the outcome by reading the receipt. Different economics under the same identity are refused, whether or not the hash matches them. The program targets the classic SPL Token program; broad Token-2022 support is not implemented or claimed. Tests run against a local validator, never a public RPC.

A Daml settlement workflow models the smallest useful flow: an operator proposes a SettlementInstruction covering one or more legs, every affected party accepts, and Settle applies all legs atomically and creates a SettlementReceipt keyed by the settlement identity. Duplicate identities, partial acceptance, and insufficient holdings are all rejected by the ledger.

The receipt records what was settled, rather than a string the client supplied. Settle takes the legs the caller believes it is settling and refuses to run unless they match the instruction, leg for leg and count included. The receipt then stores the applied legs, their count, and a canonical text encoding — settlementId|legCount|payer>payee:amount|… — that the ledger derives from those legs. It is a deterministic encoding, not a cryptographic hash: Daml SDK 2.10.6 exposes no sha256 to contract code, and the off-chain manifest covers trade ids, instrument, price ticks, and quantities that the ledger never receives, so recomputing the client's digest on the ledger is not possible and is not pretended. Reconciliation therefore compares stored economics, not a copied string. Changing an amount or a participant under the same identity is rejected, and an exact duplicate stays idempotent. What this does not prove: nothing on the ledger links the netted legs back to the individual trades, so a client that nets the wrong trades into well-formed legs would still settle.

Debit on a holding requires both the operator and the owner; the operator cannot unilaterally reduce another party's balance. Settle still works because the instruction signatories authorize the nested debit. That is the authorization change this lab makes.

What this does not do: the TypeScript venue still presents one process token that can act as every settling party, and every involved party observes the whole SettlementInstruction, including legs they are not on. A full Canton privacy or identity redesign is out of scope. Do not read the local sandbox as a confidential multi-party deployment.

The Holding template is an explicit lab placeholder for a real holding, not a token standard implementation: the Canton Network Token Standard packages are not deployed on a plain sandbox, so this model settles against its own minimal holdings and would be pointed at the deployed standard on a real network. Verified against Daml SDK 2.10.6 with the HTTP JSON API; the TypeScript venue talks to that API and treats an unreachable participant as unknown, never as failed.

cargo test --workspace                       # unit, golden, property, threaded, replay
cd apps/control-api && npm test              # control plane, projection, settlement
cd adapters/canton && daml test              # Daml settlement workflow
cd adapters/solana && anchor test            # Solana program on a local validator

What the suite actually checks:

• Reference model — a deliberately simple VecDeque order book is driven alongside the indexed one under property tests; the two must agree after every command
• Golden fixtures — recorded feeds with recorded digests, so an accidental behaviour change fails loudly instead of silently
• Decoder hardening — arbitrary bytes, every truncated prefix, and single-bit flips are property tested; four cargo-fuzz targets cover the frame, header, and output decoders. Random mutation almost never produces a correct CRC, so the raw frame target rarely gets past framing; a fourth target normalizes only the length, reserved flags, schema version, and checksum, leaving the message type, header fields, and payload to the fuzzer, so payload parsing is actually reachable. Nothing in the decoder is bypassed or reimplemented, and a unit test proves a real frame body survives that wrapper and decodes
• Threaded equivalence — a three-thread run must reproduce the single-threaded replay digests
• Invariants — reservations match live orders after every command, positions equal signed fills, cumulative fills never exceed the accepted total, output sequences never skip
• Failure injection — gateway restart, silent gateway, corrupt journal tail, duplicate output, database restart mid-flight, venue unavailable, timeout after the venue applied a batch, manifest conflict, projection run-id mismatch, output-sequence gap, and telemetry saturation

crates/protocol         Integer domain types, binary feed codec, canonical output encoding
crates/trading-core     Market view, order book, matching, risk, engine state machine, replay
crates/engine-runtime   Threads, bounded SPSC queues, journal, snapshots, latency harness
crates/control-gateway  Cold IPC boundary between the engine and the control plane
apps/simulator          Generate, replay, step, verify, and benchmark from the command line
apps/control-api        TypeScript operational API, SQLite projection, settlement outbox
adapters/solana         Anchor settlement program and its local-validator tests
adapters/canton         Daml settlement workflow and its Daml Script tests
fixtures/               Recorded feeds and their expected digests
fuzz/                   libFuzzer targets for the decoders

1. Branch from main with a descriptive name, for example feat/replace-priority-rules, fix/journal-tail-recovery, or perf/book-cancel-path.
2. Keep each change focused. One behavioural change per pull request is much easier to review than a mixed refactor.
3. Run the local checks before pushing:

   cargo fmt --all
   cargo clippy --workspace --all-targets --all-features -- -D warnings
   cargo test --workspace
   cd apps/control-api && npm run typecheck && npm run lint && npm test

4. Write a clear commit message in conventional style, for example feat(book): keep priority on a same-price decrease.
5. Open a pull request describing what changed, why, and how it was verified. Include benchmark output if the change is about performance, and say which machine produced it.
6. CI must pass. It runs formatting, cargo check, Clippy with warnings denied, the full Rust test suite, replay verification against the golden fixtures, the fuzz target build, the TypeScript typecheck, lint and tests, the Solana program and SolanaVenue tests against a local validator, and the Daml workflow plus CantonVenue tests against a local sandbox and JSON API. Venue client tests fail if the local service is missing; they do not skip. No job depends on a public RPC endpoint, a public ledger, a paid API, or a secret.
7. Squash or rebase so the merged history stays readable.

MIT © Ayush Kumar Mishra