DELPHI is a chaos engineering framework for fault injection against Linux-based systems. Originally built for Oracle Fusion Applications on OCI, it generalizes to Oracle Linux 8 and other Linux 8 variants. It operates across six chaos domains: system, network, process, Fusion/WebLogic, Oracle DB, and time/certificate manipulation.
Browser (LAN) Watson:5050 Targets (:8000)
┌──────────────┐ ┌───────────────────┐ ┌──────────────────┐
│ React UI │ HTTP │ FastAPI Backend │ HTTP │ chaosbox (8c/16g)│
│ ─────────── │ ◄──────► │ ────────────────── │ ◄──────► │ fa01 (2c/4g) │
│ • Dashboard │ │ • /api/targets │ │ ohs1 (2c/4g) │
│ • Workflows │ │ • /api/workflows │ │ opt1 (2c/4g) │
│ • Campaigns │ │ • /api/campaigns │ ├──────────────────┤
│ • React Flow │ │ • SQLite DB │ │ POST /experiment │
└──────────────┘ └───────────────────┘ │ GET /status │
│ GET /functions │
│ • chaos.sh │
└──────────────────┘
Also supports direct CLI and manifest-based execution modes.
delphi/
├── agent/ # Target-side chaos agent API (FastAPI)
│ ├── app.py # FastAPI routes, error handling
│ ├── runner.py # ExperimentRunner — subprocess lifecycle
│ ├── models.py # Pydantic models
│ └── delphi-agent.service # systemd service unit
├── ai/ # DELPHI-ai: AI-driven experiment suggestion
│ ├── rules.py # Pattern-matching rule engine
│ └── README.md # Phase 4 design doc
├── common/
│ └── schemas.py # Pydantic models: TestResult, ChaosSuggestion
├── controller/
│ ├── chaos_controller.py # Manifest-based payload controller
│ ├── agent_client.py # HTTP client for agent API
│ └── targets.conf # Target registry (name → URL)
├── core/
│ ├── chaos_controller.py # Interactive CLI (menu system)
│ ├── chaos_script/
│ │ ├── chaos.sh # Shell entry point
│ │ ├── chaos_functions.sh # All fault injection functions (38)
│ │ ├── chaos_scenario.sh # Scenario runner (sequential/parallel)
│ │ ├── net_services.txt # Domain groups for network blocking
│ │ └── scenarios/ # app.pressure, chaos.verify, db.pressure, wls.pressure
│ ├── config/ # Settings, logging, experiment logger
│ ├── lab_build/devbox/ # Vagrantfile + provision for multi-VM lab
│ ├── modules/ # Python chaos modules (ChaosResult API)
│ └── payloads/ # Multi-host incident manifests
├── scripts/
│ ├── deploy_agent.sh # Agent deployment to a single target
│ ├── deploy_all_agents.sh # Deploy agent to all 4 lab targets
│ └── install_depends.sh # dnf-based dependency installer
├── tests/ # 238 tests (pytest)
├── web/ # Web control plane (FastAPI + React)
│ ├── app.py # FastAPI app, serves React build
│ ├── database.py # SQLite layer (4 tables, WAL mode)
│ ├── models.py # Pydantic request/response models
│ ├── routers/ # targets, workflows, campaigns
│ └── static/frontend/ # React 18 + React Flow + Vite
docs/
├── delphi-architecture.md
├── delphi-engineering.md
├── delphi-roadmap.md
├── delphi-runbook.md
└── grafana_board.json # Grafana dashboard (Phase 7)
| Domain | Examples |
|---|---|
| System | CPU pressure, memory pressure, disk I/O, fill disk, OOM bomb, reboot, kernel panic |
| Network | Port blocking, IP blocking, domain blocking, latency injection (tc/netem), network I/O pressure |
| Process | CPU throttle (cpulimit), process freeze/unfreeze, service kill, observability agent kill |
| Fusion/WebLogic | Component restart, thread kill, log rotation, patch failure simulation |
| Oracle DB | Listener kill, port block, session kill |
| Time/Certs | Time skew, certificate expiry simulation, faketime execution |
- Web Control Plane — React Flow drag-and-drop workflow builder with live execution feedback
- Agent API — REST API on target boxes:
POST /api/v1/experiment, poll for results - Controller CLI —
python -m delphi.controller.chaos_controller --target chaosbox --function cpu_pressure - Interactive CLI — Menu-driven Python interface (
core/chaos_controller.py) - Payload Controller — Manifest-based multi-host parallel execution
- Shell Direct — Run
chaos.shfunctions directly on a target box
cd delphi/core/lab_build/devbox
vagrant up # all 4 targets (chaosbox, fa01, ohs1, opt1)
vagrant up chaosbox fa01 # or just specific targetsscripts/deploy_all_agents.sh # deploy to all targets
scripts/deploy_agent.sh 192.168.56.10 # or a single target
### 3. Start the Web UI
```bash
pip install -r requirements.txt
python -m delphi.web --host 0.0.0.0 --port 5050Open http://localhost:5050 — add your target, build a workflow, execute it.
cd delphi/
source .venv/bin/activate
pytest tests/ -v238 tests across all components:
| Test File | Tests | Scope |
|---|---|---|
test_chaos_modules.py |
17 | CPU, memory, disk, network modules |
test_process_state_network.py |
23 | Process, state failure, advanced network |
test_jvm_time_random.py |
26 | JVM, time, random chaos modules |
test_logging.py |
41 | Experiment logger, ChaosResult timing, sessions |
test_agent.py |
38 | Agent API, runner, models, endpoints |
test_controller.py |
23 | AgentClient, TargetRegistry, controller |
test_web.py |
70 | Database CRUD, models, API endpoints |
- OS: Oracle Linux 8 (OEL8)
- VMs (Vagrant/libvirt or VirtualBox,
generic/oracle8box):Target IP CPU RAM Role chaosbox192.168.56.10 8 16 GB Primary stress target fa01192.168.56.11 2 4 GB Fusion Apps simulation ohs1192.168.56.12 2 4 GB OHS / web tier opt1192.168.56.13 2 4 GB Optional services - Dependencies: stress, fio, iperf3, cpulimit, iproute-tc, libfaketime, bc, nc, net-tools, curl, kernel-modules-extra
| Document | Description |
|---|---|
| Architecture | System design, component map, execution flows |
| Engineering | Module APIs, test matrix, error handling |
| Runbook | Lab setup, usage examples, troubleshooting |
| Roadmap | Phased development plan |
- Never run chaos on the development machine (Watson)
kill_servicerefuses to killsshd/sshtemp_*functions auto-revert after durationkernel_panicandoom_bombare destructive — VM recovery required- Always validate with
chaos.verifyscenario first on new targets
MIT License — Copyright (c) 2025 Lindison Webb