Skip to content
BranchesTags

Repository files navigation

Agent Orchestrator v1 (Go)

A Kubernetes-inspired orchestrator for agent workloads with control-plane reconciliation, queue-based execution workers, policy gating, and production-oriented hardening.

Implemented

  • REST resources: Agent, AgentRun, Tool, GuardrailPolicy, MemoryPolicy
  • Policy decision audit endpoint: GET /policy-audits
  • Run controls: POST /agent-runs/{name}/cancel, POST /agent-runs/{name}/retry-task, GET /agent-runs/{name}/events
  • Queue operations: GET /queue/dlq, POST /queue/dlq/{id}/replay, POST /queue/dlq/{id}/discard
  • Reconcile loop + decoupled worker execution queue
  • Controller leader election via Postgres advisory transaction lock (LEADER_LOCK_ID)
  • Tenant sharding controls (SHARD_INDEX, SHARD_TOTAL) with owner-discovery endpoint
  • Admission webhook for mutating API requests (ADMISSION_WEBHOOK_URL)
  • Agent rollout strategies (canary and blue/green) via POST /agents/{name}/rollout
  • Placement and scheduling controls (requiredPool, maxConcurrentRuns)
  • Queue backends:
    • in-memory (default)
    • Redis Streams consumer groups (REDIS_ADDR) with visibility timeout redelivery and DLQ stream
  • External plugin runtime for real tool execution (PLUGIN_RUNTIME_URL, PLUGIN_RUNTIME_API_KEY)
  • Postgres normalized persistence backend when STORE_SNAPSHOT_PATH is a Postgres DSN
  • Migration tool: go run ./cmd/migrate
    • supports advisory lock (MIGRATE_LOCK_ID) and dry-run (MIGRATE_DRY_RUN=true)
  • Auth and authorization:
    • API key (X-API-Key)
    • JWT HS256 (Authorization: Bearer ..., JWT_HS256_SECRET)
    • RBAC scopes (runs:read, runs:write, runs:approve, runs:cancel, runs:retry, runs:replay, policy:audit:read, policy:read, policy:write, config:read, config:write, queue:dlq:read, queue:dlq:write, admin)
  • Reliability controls:
    • step checkpoints + DLQ
    • tool retry classification and circuit breaker
    • request idempotency (X-Idempotency-Key)
  • Metrics (/metrics) and OpenAPI (/openapi.yaml)
  • Opaque pagination for list endpoints via pageToken and X-Next-Page-Token
  • Optional OTLP tracing export (OTEL_EXPORTER_OTLP_ENDPOINT)
  • Example Prometheus alert rules and runbooks under /Users/mchenetz/git/ao/monitoring/alerts.yaml and /Users/mchenetz/git/ao/docs/runbooks/
  • CI workflows for tests, vuln/security checks, and release image publish

Run locally (in-memory queue + file store)

API_KEY=dev-secret DEFAULT_TENANT=acme go run ./cmd/server

CLI (aoctl)

go run ./cmd/aoctl --url http://localhost:8000 --api-key dev-secret --tenant acme health
go run ./cmd/aoctl --url http://localhost:8000 --api-key dev-secret --tenant acme agents list
go run ./cmd/aoctl --url http://localhost:8000 --api-key dev-secret --tenant acme runs create --name run-1 --agent-ref support-agent --goal "Send customer email"
go run ./cmd/aoctl --url http://localhost:8000 --api-key dev-secret --tenant acme runs events --name run-1 --limit 50
go run ./cmd/aoctl --url http://localhost:8000 --api-key dev-secret --tenant acme apply -f job.yaml

aoctl also supports a kubeconfig-style context file at ~/.ao/config.yaml. See /Users/mchenetz/git/ao/docs/aoctl.md.

Run locally with Postgres + Redis

export DATABASE_URL='postgres://orchestrator:orchestrator@localhost:5432/orchestrator?sslmode=disable'
go run ./cmd/migrate
API_KEY=dev-secret JWT_HS256_SECRET=dev-jwt-secret DEFAULT_TENANT=acme STORE_SNAPSHOT_PATH="$DATABASE_URL" REDIS_ADDR=localhost:6379 go run ./cmd/server

External Plugin Runtime

Run plugin runtime:

cp /Users/mchenetz/git/ao/deploy/plugin-runtime/plugins.yaml.example ./plugins.yaml
PLUGIN_RUNTIME_CONFIG=./plugins.yaml PLUGIN_RUNTIME_API_KEY=plugin-secret go run ./cmd/plugin-runtime

Point orchestrator to it:

PLUGIN_RUNTIME_URL=http://localhost:9000 PLUGIN_RUNTIME_API_KEY=plugin-secret API_KEY=dev-secret DEFAULT_TENANT=acme go run ./cmd/server

Create a tool routed through runtime:

curl -sX POST localhost:8000/tools \
  -H 'content-type: application/json' \
  -H 'X-API-Key: dev-secret' \
  -H 'X-Tenant-ID: acme' \
  -d '{"name":"email-send","runtime":"plugin-runtime","authRef":"bearer:my-token","timeoutMs":15000}'

Run with Docker Compose

docker compose up --build

Compose services are configured with restart: unless-stopped so API/DB/Redis stay running after crashes or reboots. For host-managed always-on services, use templates:

  • systemd: /Users/mchenetz/git/ao/deploy/systemd/ao-server.service
  • launchd (macOS): /Users/mchenetz/git/ao/deploy/launchd/com.mchenetz.ao.server.plist

Linux systemd (always-on)

Build and install:

go build -o agent-orchestrator ./cmd/server
sudo /Users/mchenetz/git/ao/deploy/systemd/install.sh ./agent-orchestrator

Configure env:

sudo cp /Users/mchenetz/git/ao/deploy/systemd/ao-server.env.example /etc/ao/ao-server.env
sudo vi /etc/ao/ao-server.env

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now ao-server
sudo systemctl status ao-server

E2E With Postgres + Redis (Compose)

docker compose up -d postgres redis
DATABASE_URL='postgres://orchestrator:orchestrator@localhost:5432/orchestrator?sslmode=disable' go run ./cmd/migrate
POSTGRES_TEST_DSN='postgres://orchestrator:orchestrator@localhost:5432/orchestrator?sslmode=disable' REDIS_TEST_ADDR='localhost:6379' go test ./internal/orchestrator -run 'TestPostgresRedisFlow|TestLeaderLockContentionPostgres|TestShardedMultiServerPostgresRedisFlow' -count=1

RBAC Scopes

  • Run read/write: runs:read, runs:write
  • Run controls: runs:approve, runs:cancel, runs:retry, runs:replay
  • Policy/config: policy:read, policy:write, policy:audit:read, config:read, config:write
  • Queue ops: queue:dlq:read, queue:dlq:write
  • Federation owner lookup: admin

JWT example payload for rollout + run operations:

{
  "sub": "platform-operator",
  "tenant_id": "acme",
  "scope": "config:write runs:write runs:read runs:approve runs:cancel runs:retry runs:replay queue:dlq:read queue:dlq:write"
}

JWT example payload for federation owner lookup:

{
  "sub": "cluster-admin",
  "role": "admin"
}

Additional Runbooks

  • Rollouts: /Users/mchenetz/git/ao/docs/runbooks/rollouts.md
  • Admission webhook failures: /Users/mchenetz/git/ao/docs/runbooks/admission-webhook.md
  • Spec traceability checklist: /Users/mchenetz/git/ao/docs/spec-traceability.md

Full Quick Start

  • /Users/mchenetz/git/ao/docs/quickstart.md
  • CLI reference and examples: /Users/mchenetz/git/ao/docs/aoctl.md

Quick API flow

curl -sX POST localhost:8000/guardrail-policies \
  -H 'content-type: application/json' \
  -H 'X-API-Key: dev-secret' \
  -H 'X-Tenant-ID: acme' \
  -H 'X-Idempotency-Key: pol-1' \
  -d '{"name":"standard","rules":[{"expr":"tool == '"'"'email-send'"'"' && recipient_external == true","action":"require_approval"}]}'

curl -sX POST localhost:8000/agents \
  -H 'content-type: application/json' \
  -H 'X-API-Key: dev-secret' \
  -H 'X-Tenant-ID: acme' \
  -H 'X-Idempotency-Key: agent-1' \
  -d '{"name":"support-agent","instructionsRef":"s3://agent.md","tools":["email-send"],"guardrailPolicyRef":"standard"}'

curl -sX POST localhost:8000/agent-runs \
  -H 'content-type: application/json' \
  -H 'X-API-Key: dev-secret' \
  -H 'X-Tenant-ID: acme' \
  -H 'X-Idempotency-Key: run-1' \
  -d '{"name":"run-1","agentRef":"support-agent","goal":"Send customer email","constraints":{"requireApprovalFor":["external_send"]}}'

curl -s localhost:8000/agent-runs/run-1 -H 'X-API-Key: dev-secret' -H 'X-Tenant-ID: acme'
curl -sX POST localhost:8000/agent-runs/run-1/approvals -H 'content-type: application/json' -H 'X-API-Key: dev-secret' -H 'X-Tenant-ID: acme' -d '{"decision":"approved","actor":"oncall"}'
curl -s localhost:8000/policy-audits?run=run-1 -H 'X-API-Key: dev-secret' -H 'X-Tenant-ID: acme'

About

A Kubernetes-inspired orchestrator for agent workloads with control-plane reconciliation, queue-based execution workers, policy gating, and production-oriented hardening.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors