To compile khaos and its eBPF components, make sure your system has the following dependencies:

sudo apt update && sudo apt install -y \
  clang \
  llvm \
  gcc \
  make \
  pkg-config \
  build-essential \
  libelf-dev \
  zlib1g-dev \
  libzstd-dev \
  libssl-dev \
  linux-headers-$(uname -r) \
  linux-tools-$(uname -r)

Next, make sure the libbpf submodule is initialized:

git submodule update --init --recursive

Build and install libbpf:

cd libbpf/src
sudo make install

Setup headers (amd64/x86):

sudo apt install linux-headers-`uname -r`

Create a symlink for headers:

sudo ln -s /usr/include/x86_64-linux-gnu/asm /usr/include/asm

Setup headers (arm): Use the setup_arm64_headers.sh script.

You can then compile the full project:

cd ../..  # return to root if in libbpf
make # or make ARCH=arm for ARM builds

sudo ./khaos <fault_type> <pid>

You may also recover a fault:

sudo ./khaos --recover <fault_type>

🐳 Docker Image (Quick Start)

If you prefer to run Khaos inside a container, we've provided a Docker image setup.

docker build -t khaos:latest .

# Create and use buildx builder
docker buildx create --name khaos-builder --driver docker-container --bootstrap
docker buildx use khaos-builder

# Build and push multi-arch image
docker buildx build --platform linux/amd64,linux/arm64 -t your-registry/khaos:latest --push .

Make sure you have compiled the Khaos binary and required .bpf.o and .skel.h files in the build context.
Run the Container

To inject a fault:
```bash
sudo docker run --rm -it \
  --privileged \
  --pid=host \
  --network=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -v /proc:/host/proc:ro \
  ghcr.io/xlab-uiuc/khaos:latest <fault_type> <pid>

To recover:

sudo docker run --rm -it \
  --privileged \
  --pid=host \
  --network=host \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -v /proc:/host/proc:ro \
  ghcr.io/xlab-uiuc/khaos:latest --recover <fault_type>

⚠️ You must run the container with --privileged and proper mounts to enable eBPF functionality.

Check the kernel and container prerequisites with a temporary probe whose empty PID map prevents it from changing any syscall result:

sudo docker run --rm \
  --privileged \
  --pid=host \
  -v /sys/fs/bpf:/sys/fs/bpf \
  -v /sys/kernel/btf:/sys/kernel/btf:ro \
  ghcr.io/xlab-uiuc/khaos:latest --check

The PID supplied to Khaos must be visible in the Khaos container's /proc, so the container must use the host PID namespace. Khaos reads that task's NSpid chain and /proc/<pid>/ns/pid identity automatically, then filters in the target task's active PID namespace. This also works when the target is in a nested Docker or Kubernetes PID namespace. To inspect the translation without injecting a fault, run /khaos/khaos --resolve-pid <visible-pid>.

examples/docker-compose.yaml provides a long-running privileged sidecar suitable for Docker Compose environments, including Harbor tasks. Keep Docker control in the trusted task harness; the agent container does not need the Docker socket. The harness can inject and recover faults with:

docker compose exec -T khaos /khaos/khaos read_error <pid>
docker compose exec -T khaos /khaos/khaos --recover read_error <pid>

On kind, deploy the same image as a privileged hostPID DaemonSet. PIDs read from the kind node's /proc are matched in that node's PID namespace, even when the kind node itself is a Docker container.

make tests

You'll find a mixture of Python and C programs that are used for testing the various faults.

sudo ./khaos read_error 1234

• Hooks into read() (__arm64_sys_write).
• Injects error -5 (EIO, Input/Output error).
• Affects process 1234—any read() calls from this process will fail.

You can run the demo program python3 read_demo.py and test with it's PID.

Before running khaos, ensure your system is properly configured to allow BPF programs to execute.

Verify that your kernel supports BPF by checking:

grep CONFIG_BPF_SYSCALL /boot/config-$(uname -r)

Expected output:

CONFIG_BPF_SYSCALL=y

If this is not set, you may need to recompile your kernel with BPF support.

BPF relies on DebugFS and Tracing for attaching probes. Ensure DebugFS is mounted:

sudo mount -t debugfs none /sys/kernel/debug

Check for available tracing functions:

ls /sys/kernel/debug/tracing/

BPF programs require a sufficient locked memory limit. Increase it:

ulimit -l unlimited

To make this persistent, add the following line to /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited

By default, Linux may restrict unprivileged BPF usage. Check the current setting:

cat /proc/sys/kernel/unprivileged_bpf_disabled

If the output is 1 or 2, allow unprivileged BPF by running:

sudo sysctl -w kernel.unprivileged_bpf_disabled=0

To make this persistent:

echo "kernel.unprivileged_bpf_disabled=0" | sudo tee -a /etc/sysctl.conf
sudo sysctl --system

Your system needs kernel headers that match your running kernel. Install them with:

sudo apt install linux-headers-$(uname -r)

If you're on ARM64, make sure the correct headers are linked:

sudo ln -s /usr/src/linux-headers-$(uname -r)/arch/arm64/include/generated/uapi/asm /usr/include/asm

Prompt the following command into your shell to clone the repository and its submodules:

git clone https://github.com/xlab-uiuc/khaos.git
cd khaos
git submodule update --init --recursive

To build both static libbpf.a and shared libbpf.so:

pushd libbpf/src
sudo make install

Then compile khaos: Problems with aarch64 and arm64 architecture? Try the guide below then make ARCH=arm command again

popd
make #ARCH=arm for arm compilation, default is x86

You will get a khaos binary file.

WSL2 does not support eBPF by default, you will have to compile a custom kernel if you intend to continue using it.

If you're running Ubuntu like I am, you might have to disable AppArmor for the program to work.

If you're into problems using make ARCH=arm with one of the following errors:

In file included from khaos.bpf.c:1:
In file included from /usr/include/linux/bpf.h:11:
In file included from /usr/include/linux/types.h:5:
In file included from /usr/include/asm/types.h:1:
In file included from /usr/include/asm-generic/types.h:7:
/usr/include/asm-generic/int-ll64.h:12:10: fatal error: 'asm/bitsperlong.h' file not found
   12 | #include <asm/bitsperlong.h>
      |          ^~~~~~~~~~~~~~~~~~~

1 error generated.
make: *** [Makefile:24: khaos.bpf.o] Error 1 

You might want to run the sh script below to resolve the problem:

sudo apt update
sudo apt install linux-headers-$(uname -r) libc6-dev build-essential
chmod +x setup_arm64_headers.sh
./setup_arm64_headers.sh

Most BPF operations require root privileges. Always run khaos with sudo:

sudo ./khaos <syscall_name> <error_code> <pid>

The first step to adding a new fault is to identify the syscall we want to inject a fault on. You can find relevant syscalls by using the strace tool to see the syscalls used by a particular program:

Here’s a polished and expanded version of your README.md section on adding new faults, including examples and detailed steps:

Adding a new fault to Khaos involves identifying the target syscall, choosing the appropriate error code, and registering the fault in the Khaos fault registry. Here's a step-by-step guide:

To determine which syscall a program is using (and might fail on), use strace to trace its system calls.

If you have a Python script read_demo.py that reads a file:

strace -e trace=read,open,write -p <PID>

Or, to launch the program with tracing from the start:

strace -e trace=all python3 read_demo.py

You will see output like:

openat(AT_FDCWD, "test_file.txt", O_RDONLY) = 3
read(3, "Hello world\n", 1024)             = 12

This tells you the program uses the openat and read syscalls — candidates for fault injection.

Once you identify the syscall, consult its manual page to learn what error codes it may return.

Run:

man 2 read

Look for the ERRORS section. Example for read(2):

EAGAIN      The file descriptor refers to a file other than a socket and has been marked nonblocking...
EIO         A low-level I/O error occurred while reading from the disk.
EBADF       fd is not a valid file descriptor or is not open for reading.

Pick an error code that meaningfully simulates a hardware-related fault. For example:

• EIO (-5) simulates a disk read failure
• ENOSPC (-28) simulates a full disk
• ENOMEM (-12) simulates memory exhaustion

Open khaos.c and locate the fault_registry[] struct:

static struct fault_entry fault_registry[] = {
    {"read_error", "read", -5},       // Injects EIO
    {"write_error", "write", -28},    // Injects ENOSPC
    ...
};

Add a new line for your fault:

{"open_error", "openat", -13},   // Injects EACCES

Make sure:

• The name is unique (e.g., open_error)
• The syscall name matches exactly what’s used in the kernel (strace output will guide this)
• The error code is negative