Embrace the Tumult

Rust-native chaos engineering, end to end. Author experiments in the web UI, approve them behind risk-tiered quorums, run them with real telemetry — and keep every result as compliance-grade evidence.

$ curl -sSL https://raw.githubusercontent.com/mwigge/tumult/main/install.sh | sh

click to copy

$ docker pull ghcr.io/mwigge/tumult:latest

click to copy

Get Started GitHub Blog

1,300+

Unit Tests

MCP Tools

Plugins

Chaos Actions

T0–T3

Approval Risk Tiers

Unsafe Blocks (audited — process-group kills)

Compliance Frameworks

One Platform, End to End

Tumult is a single platform, not a pile of tools. Every part feeds the next, and the loop closes when what you learn shapes the next experiment.

Author

Catalog-driven wizard in the web UI, or TOON from the CLI

→

Approve

Risk tiers T0–T3: quorum, TTL, content-hash pin

→

Run

Real OpenTelemetry traces on every action

→

Store

OTLP ingest into the Kronika DuckDB lake

→

Explore

Web UI, SQL, or plain questions in natural language

→

Report

Compliance evidence as document-controlled PDFs

→

Recommend

AI agents over MCP suggest the next experiment

↺ Recommendations feed back into the next experiment definition.

The Parts of the Platform

Each piece does one job and hands off to the next. Here's what fits where.

Author & run

Web UI Authoring & CLI

Pick a fault from the live plugin catalog and the Author wizard scaffolds validated TOON — one click into a registered, runnable definition. Or write TOON by hand and run it from one static binary: 16 plugins, 91 chaos actions across PostgreSQL, MySQL, Redis, Kafka, Docker, Podman, Pumba network chaos, Kubernetes, SSH, and CPU/memory/IO stress.

Govern

Approvals, E-Stop & Audit

Every run is classified into a risk tier (T0–T3) at request time. Gated runs wait behind a hash-pinned, quorum- and TTL-bound approval with segregation of duties — the requester can never approve their own run. A two-step e-stop and a stop-all kill switch halt anything mid-method, and every action lands in a hash-chained, tamper-evident audit trail.

Store & explore

Kronika Lake & Analytics

The daemon (tumultd) ingests OTLP traces, metrics, and logs around the clock into an embedded DuckDB store — Kronika — then serves the web UI from the same binary. KPIs, experiment history, log/trace/metric explorers, per-domain score rollups, parquet export. Or just ask a question in plain language over the store.

Prove it

Reports & Compliance

Map experiments to DORA (EU 2022/2554), NIS2, PCI-DSS 4.0, ISO 22301, ISO 27001, SOC 2, and Basel III via tumult compliance. R1 executive digests and R3 game-day reports sit on top of any run; the R2 evidence pack renders for DORA, NIS2, ISO 27001, and SOC 2 — PCI-DSS 4.0, ISO 22301, and Basel III are mapping-only. All as A4 PDFs with document control and hash-chain verification.

Close the loop

MCP Server & Agents

40 MCP tools over stdio or Streamable HTTP, with tool annotations, structured output schemas, and tumult:// resources. Any MCP-compatible agent can discover plugins, run experiments, query the analytics lake, and recommend what to test next — and Tumult can inject faults into agentic AI systems themselves.

Built to Run Anywhere

Single Binary

One static binary — no Python, no runtime dependencies. The daemon embeds the lake, the schedulers, and the web UI in a single tumultd; the CLI is another. Pre-built for macOS (x86_64/aarch64) and Linux (x86_64 gnu/musl, aarch64 musl).

Native OpenTelemetry

Every experiment emits real OTel spans across SSH, Kubernetes, plugins, baselines, analytics, and MCP dispatch. Traces show up in Jaeger, SigNoz, or any OTLP backend — no configuration.

Memory Safe

Safe Rust across all 32 crates, with exactly two audited unsafe blocks for process-group kills. Zero production .unwrap(). Clippy pedantic enforced. cargo-audit on every commit.

GameDay Orchestration

Coordinated campaigns of experiments with shared load, resilience scoring, and compliance article mapping. One command: 4 experiments, DORA/NIS2 evidence, audit-ready journal.

Docker Images

Pre-built on GHCR — no Rust toolchain needed. docker pull ghcr.io/mwigge/tumult. Composable bundles: infra, observability, MCP server, agent fleet. Full e2e in one command.

SigNoz Observability

OTel Collector (contrib) with OTLP + Arrow receivers, span-to-metrics APM, host metrics, Prometheus. Traces flow to SigNoz — dashboards, alerting, log aggregation. No custom build needed.

See It in Action

↗ share this demo: tumult.rs/watch

GameDay: One Command, Evidence Mapped

$ ./scripts/gameday-demo.sh Targets: ✓ postgres ✓ redis ✓ kafka MCP Session: 2609892c-8e73-4587-b96a-56eaf6020ec6 Discovering chaos capabilities: Plugins: 16 | Actions: 91 Running Q2 PostgreSQL Resilience GameDay... Compliance: DORA EU 2022/2554 Art. 11, 24, 25 | NIS2 GameDay: Q2 PostgreSQL Resilience Programme Status: COMPLIANT Resilience Score: 1.00 #1 [PASS] Connection kill under load (2197ms) #2 [PASS] Container pause — total outage (7402ms) #3 [PASS] CPU stress — resource pressure (9331ms) #4 [PASS] Memory stress — resource pressure (9305ms) Score: Pass 1.00 | Recovery 1.00 | Load 1.00 | Compliance 1.00 Store: 67 experiments, 244 activities (DuckDB)

Compliance reports are evidence summaries, not legal or audit attestations.

The Platform, Running

Every screen below is the Tumult web UI on the seeded demo stack (docker compose -f docker/docker-compose.kronika.yml up -d) — from authoring to an approved, executed, e-stopped and reported run. The full click path is in the platform walkthrough.

Tumult overview dashboard — experiment KPIs, hypothesis pass and deviation rates, fault breakdown over the selected window

OverviewResilience posture at a glance: hypothesis pass rate, deviation rate, experiments per day, fault breakdown — computed from the store, per window.

Author page — the live plugin fault catalog with network and database actions and probes, one click to scaffold an experiment

Catalog-driven authoringEvery fault and probe from the live plugin catalog, searchable and documented — pick one and the wizard scaffolds a validated experiment.

Approvals queue — an approver reviews a pending T2 run with its content-hash pin

Approval workflowsRisk-tiered runs (T0–T3) park behind a quorum with content-hash pinning and segregation of duties — the requester can never approve their own run.

Runs page — state filters, an aborted run with rollback completed, and the Halt all kill switch

Stop-all kill switchState filters over the run queue, and a two-click Halt all that stops every active run and unwinds rollbacks — with the actor in the audit trail.

Run detail — deviated run with live telemetry waterfall, consumed approval chain and hash-chained audit trail

Run detailLive telemetry waterfall, the consumed approval chain with the approver's note, and a tamper-evident hash-chained audit trail — one page, whole story.

Aborted run after two-step e-stop — fault span halted mid-method, rollback completed

Two-step e-stopStop a run mid-method: the fault span halts, rollbacks unwind to a clean state, and the stop joins the audit trail with its actor.

GameDays page — a registered campaign and the campaign TOON registration form

GameDaysCoordinated campaigns registered from campaign TOON, executed under the same guardrails, scored and mapped to compliance articles.

Reports page — a generated R2 DORA evidence pack with SHA-256 and PDF download

Evidence packsR1/R2/R3 compliance reports generated from the store as document-controlled PDFs — the R2 pack includes the approval chain (SOC 2 CC8.1).

Architecture

Native Observability

Every experiment produces a full trace — hypothesis, actions, probes, rollbacks — visible in SigNoz, Jaeger, or any OTLP backend. No configuration required.

SigNoz showing Tumult experiment trace waterfall

Ready to Build Resilience?

One command to install. One script to run a full GameDay with DORA/NIS2 evidence mapping.

View on GitHub Quickstart GameDay Guide