Kubernetes Agent Runtime Measurement Architecture
Modular Framework for Evaluating AI Agents in Kubernetes
📺 Watch the 2-minute demo screencast
About | Why KARMA | Roadmap | Quick Start | Repo Map | Profiles | Developer Guide | License
KARMA is a framework for evaluating AI agents on realistic Kubernetes operations tasks.
Instead of treating each benchmark as a one-off script, KARMA models tasks as reusable stages that can be assembled into multi-stage workflows. Each stage prepares the environment, lets the agent act on the live system, verifies the result, and records artifacts for later analysis.
The point is simple: isolated tasks only tell part of the story. Real operations work is stateful. Changes stack up. Fixes in one step can quietly break something earlier. KARMA is built to evaluate agents in that setting.
The suite ships 92 composable cases and 38 reversible adversary scenarios across eight systems — RabbitMQ, MongoDB, CockroachDB, Elasticsearch, Nginx, Ray, Spark, and a demo service — which recombine into hundreds of example workflows.
KARMA is built around a few ideas that matter in practice:
- Reusable building blocks. A testcase defines one operational task with setup, prompt, verification, and cleanup.
- Stateful workflows. Stages run in sequence against preserved system state, so later actions can help or hurt earlier outcomes.
- Deterministic correctness checks. Each stage has explicit verification, with optional workflow-level regression sweeps to catch cross-stage breakage.
- Probe/apply/verify setup. Testcases are broken into smaller resource-level precondition units, each with its own
probe,apply, andverifycycle. That makes cases much more chainable because a later workflow stage can reuse the parts of the environment that are already correct instead of replaying a full reset. - Trajectory capture. Runs produce logs, traces, snapshots, and optional judge artifacts so you can inspect how the agent behaved, not just whether it passed.
This makes KARMA useful for long-horizon evaluation, regression analysis, and safety-oriented agent testing in real infrastructure environments.
KARMA already supports horizontal composition: building longer workflows from reusable stages.
But many hard real-world tasks are not just longer workflows. They are a base task plus environmental adversaries such as drift, permission constraints, and competing controllers.
That is the next direction for KARMA:
- Extend KARMA beyond Kubernetes
- Add layered environmental adversaries to workflows
- Python 3.11+
kubectlon yourPATH, pointing at a reachable cluster (a local kind cluster works)- a kubeconfig at
~/.kube/config(or setKUBECONFIG)
pip install -e .
python3 main.pyThen open:
http://localhost:8080
The UI is the easiest way to browse services, inspect workflows, run cases, and generate the equivalent CLI commands.
./scripts/setup-cluster.sh --provider kindRun a single case:
python3 orchestrator.py run-case demo configmap-update \
--agent cli_runner --sandbox localRun a multi-stage workflow:
python3 orchestrator.py run-workflow workflows/suite/workflow-demo.yaml \
--agent cli_runnerJudge a completed run (LLM-as-Judge):
python3 orchestrator.py judge runs/<run_id>List the available agents:
python3 orchestrator.py info --agentsFor a local debug run that should reuse a specific namespace, a workflow can declare a debug-only alias mapping:
spec:
debug:
namespace_overrides:
default: karma-debugActivate it with python3 orchestrator.py run-workflow <workflow.yaml> --debug.
The mapping key is a workflow namespace alias (for example, default), and the
value is the Kubernetes namespace name. Without --debug, KARMA continues to
use its normal per-run namespaces. KARMA leaves creation, labeling, and cleanup
of the debug namespace to you. Case setup and reset commands still run and may
change or recreate resources in that namespace. Use a namespace intended for
testing.
pip install -e ".[dev]" # installs pytest and the test dependencies
pytest tests/unit # fast, no cluster required
pytest tests/integration # requires a reachable cluster-
main.pyHTTP server and web UI (served athttp://localhost:8080). -
orchestrator.pyHeadless CLI entrypoint forrun-case,run-workflow,run-batch,judge, andinfo. -
karma/The framework package: case/workflow/prompt definitions, the Kubernetes environment and transport (kubectl proxy), the adversary lifecycle, the runtime execution core, the oracle and evidence collection, and the LLM-as-judge pipeline. -
karma/agents/Agent adapters and container definitions (cli_runner,react,claude_code,codex,copilot,api). -
cases/tests/Benchmark corpus. Each testcase lives undercases/tests/<service>/<case>/test.yaml. -
cases/common/Shared case helpers for RabbitMQ, Ray, and Spark. -
workflows/Multi-stage workflow definitions built from reusable testcases. -
adversaries/Adversary scenarios (drift, permission constraints, competing controllers) injected into workflows. -
webui/The browser UI — plain HTML/CSS/JS with no build step. -
docs/Developer runbooks, prompt templates, and design notes. -
tests/tests/unit(fast, no cluster) andtests/integration(require a live cluster).
A profile is a reusable YAML preset of run flags (agent, sandbox, timeout, params, …) so you don't have to repeat them on every invocation. Pass one with --profile to run-case, run-workflow, or run-batch:
python3 orchestrator.py run-workflow workflows/suite/workflow-demo.yaml \
--profile my-profile.yamlNote: the LLM-as-Judge also uses rubric overlays (e.g. judge_base.yaml), which are a separate concept — they tune scoring, not execution.
If you want to understand or extend the repo, start here:
-
docs/developer/adding-a-test-case.mdHow to author reusable, workflow-safe testcases. -
docs/developer/kind-cluster-setup.mdSetting up a local Kind cluster. -
docs/developer/persistent-agent-sessions.mdHow an agent session persists across stages of a workflow. -
docs/developer/composition-failure-patterns.mdCommon failure classes in long, composed workflows.
If you are new to the codebase, a good path is:
- Skim the Repo Map above and the guides in
docs/developer/. - Open a workflow in
workflows/and a testcase incases/tests/<service>/<case>/test.yaml. - Trace
orchestrator.pyintokarma/interfaces/cli/main.pyandkarma/runtime/workflow.py. - Run a workflow and inspect the artifacts under
runs/.
KARMA is released under the MIT License.
If you use KARMA in your research, please cite:
@unpublished{karma2026,
title = {KARMA: A Composable Framework for Evaluating LLM Agents on
Stateful Kubernetes Operations},
author = {Ouyang, Junhan and Liu, Shaw and Wang, Jinlang and Zhou, Jiawei},
year = {2026},
note = {Under review at EMNLP 2026 System Demonstrations}
}