harnessc

Compile a specialized task harness from SOPs, tools, examples, and rubrics.

If you have a narrow task where a general agent is too expensive, too variable, or too loose with policy gates, harnessc helps turn the task into a small auditable runtime. The harness keeps the workflow deterministic and uses smaller, cheaper model calls only for the parts that need language judgment.

That is the value prop: task-specific performance with lower runtime cost. You spend intelligence once at build time, then run a constrained harness that can use cheap model routes, cached calls, pure functions, and deterministic tool gates.

• Evidence So Far
• Related Writing
• The Core Idea
• How It Works Internally
• Quick Start
• Build A Harness For Your Own Task
• Current Harness Inventory
• Docs Map
• Repository Tour
• Status
• License

The first head-to-head numbers are from local dev splits, using the same task packets, tool fixtures, rubrics, and strict output metrics for the compiled harness and a headless general-agent baseline.

These are not universal benchmark claims. They are evidence for the repo's target shape: SOP-bound tasks with known tools, policy gates, strict output contracts, and enough messy language that a plain workflow is not enough. The baseline is intentionally strict: if a general agent misses the exact state, tool argument, or gate contract, it fails. That exactness is what the harness sells.

See reports/eva-expansion-report.md and reports/phases-4-5-report.md for caveats, latency notes, and preserved run artifacts.

The product/research thread behind this repo is developed in these Enchiridion Labs articles:

• Specialized Harness Engineering introduces the harness thesis: the model supplies local judgment, the harness supplies structure.
• Proceda Voice explains the SOP-driven voice-agent setting that motivated the dialogue packets.
• SOP execution engine design describes the architecture for turning SOPs into executable state machines.
• SOP-Bench results shows the broader pattern that structured SOP execution can let smaller models outperform larger general agents on the target task.

General agents are flexible, but for policy-bound enterprise tasks they often blend too many responsibilities into one prompt: workflow control, policy recall, tool eligibility, argument construction, response phrasing, and audit. That makes them harder to review and more expensive to run.

Classical workflows have the opposite problem: they are cheap and auditable, but they expect clean fields. They break when a user answers out of order, corrects themself, gives too much information in one turn, or pressures the system to skip a confirmation.

harnessc tests a third shape:

• express the task as a declarative HarnessSpec;
• run that spec on a policy-free kernel;
• keep all tool gates, transitions, merge rules, audit events, and output contracts deterministic;
• use LLMs only in declared judgment slots with typed output schemas;
• validate generated specs, prompts, and pure functions before execution;
• evaluate the result against splits, genericity perturbations, adversarial dialogue simulation, stateful sandboxes, and a general-agent baseline.

The compiler metaphor:

The internals walkthrough is docs/design.md. It explains the compiler pipeline, the HarnessSpec IR, the policy-free kernel, judgment-slot execution, the validator wall, the eval gauntlet, the hill-climber, and the freeze artifact.

If you want the shortest conceptual version first, read docs/overview.md. If you want to author or inspect the actual spec language, read docs/spec-guide.md.

Install dependencies and run the release checks:

uv sync
make check

Run the validator wall replay:

uv run python scripts/run_wall.py

Run an existing deterministic harness:

uv run python scripts/run_gauntlet.py dangerous_goods --splits dev,holdout --genericity dev,holdout

Run an existing generated dialogue harness:

uv run python scripts/run_gauntlet.py clinic_appointment_scheduling \
  --harness-dir harnesses_generated/clinic_cold_compile \
  --splits dev,holdout,genericity

Today, harnessc is a repo workflow, not a polished package or hosted service. The happy path is to add a task packet, compile it, evaluate it, then run the generated harness through the kernel.

Start by copying the shape of packets/clinic_appointment_scheduling. A packet should contain:

packets/<task_id>/
  task.yaml                  # mode, metrics, budgets, source, split paths
  representation.md          # what the packet does and does not prove
  rubric.md                  # scoring rules and success criteria
  policy/*.md                # SOPs and policy docs
  tools/tools.json           # tool schemas, fixture strategies, irreversibility
  examples/train.jsonl
  examples/dev.jsonl
  examples/holdout.jsonl
  examples/edge_cases.jsonl
  examples/counterexamples.jsonl
  notes/*.md                 # optional known failures or domain notes

Minimum useful packet: train, dev, rubric.md, representation.md, task.yaml, policy docs, and any tools the task can call. Holdout, counterexamples, and edge cases are what make the claim credible rather than just green on examples.

Check that the packet loads:

uv run python - <<'PY'
from compiler.packets import load_packet
packet = load_packet("packets/<task_id>")
print(packet.metadata.id, sorted(packet.splits))
PY

There are two model surfaces:

• Build-time compiler backend: scripts/compile_task.py invokes the claude CLI in headless mode. Make sure Claude Code is installed and authenticated. You can pass --model <model-name> to override its default.
• Runtime judgment slots: generated specs use Gemini by default. Set one of these before running LLM-backed harnesses:

export HARNESSC_GEMINI_API_KEY="..."
# fallback names also supported by the router:
export GEMINI_API_KEY="..."
export GOOGLE_API_KEY="..."

Specs can name an explicit key with:

secret_ref: env:HARNESSC_GEMINI_API_KEY

Operator-owned runtime config, such as a pinned current date, can be passed at compile time with --config-json or stored in the harness config.json.

uv run python scripts/compile_task.py <task_id> \
  --out-name <task_id>_v1 \
  --config-json '{"current_date": "2026-07-07"}' \
  --splits train,dev,holdout,edge_cases,counterexamples \
  --genericity dev,holdout

The compiler writes the harness to:

harnesses_generated/<task_id>_v1/

It also preserves build attempts, prompts, replies, wall rejections, and cost under:

runs/compile/<task_id>_v1_<timestamp>/

If the final wall rejects the harness, treat that as useful feedback. The common fixes are underspecified slots, missing coverage for counterexamples, literal values copied from examples, or pure functions that exceed the allowed surface.

Run the generated harness against packet splits:

uv run python scripts/run_gauntlet.py <task_id> \
  --harness-dir harnesses_generated/<task_id>_v1 \
  --splits dev,holdout,edge_cases,counterexamples \
  --genericity dev,holdout

Compare it with the general-agent baseline:

uv run python scripts/run_baseline.py <task_id> \
  --harness-dir harnesses_generated/<task_id>_v1 \
  --splits dev

The baseline command writes runs/<task_id>/baseline_<timestamp>/economics.json with accuracy, cost/case, latency/case, and the harness-over-baseline cost ratio.

For now, the direct runtime interface is Python. Dialogue tasks pass turns; batch tasks pass whatever fields the spec declares in input_bindings.

uv run python - <<'PY'
import json
from compiler.packets import load_packet
from kernel.harness import Harness

task_id = "<task_id>"
packet = load_packet(f"packets/{task_id}")
harness = Harness(
    f"harnesses_generated/{task_id}_v1",
    packet.tools,
    config={"current_date": "2026-07-07"},
)

result = harness.run({
    "input": {
        "turns": [
            "Hi, I need to change my appointment to next Tuesday morning."
        ]
    }
})

print(json.dumps(result["output"], indent=2))
PY

For multi-turn dialogue simulation, look at gauntlet/session.py and the existing packet examples. For production use, you would wrap kernel.harness.Harness behind your own service boundary and replace fixture tools with live executors.

The implemented pipeline covers Phases 0-5 from the design plan:

• dangerous_goods: deterministic batch harness, 40/40 on carried-over splits.
• eva_itsm_room_policy: dialogue harness, 18/18 and formally frozen across 14 release surfaces.
• fnol_auto_insurance: dialogue harness, 12/12 for the hand-authored harness; generated recompile currently records an 11/12 residual.
• clinic_appointment_scheduling: cold-compiled generated packet, 16/16 plus sandbox and simulator coverage.
• eva_airline_rebook: generated EVA packet, 15/16 with an adversarial confirmation-gate residual preserved in the packet.
• eva_medical_hr_fmla: generated EVA packet, 12/13 with a documented spec+prompt residual.

Start here if you are new:

• docs/overview.md explains the problem, mental model, compile loop, runtime shape, and evidence trail without assuming prior repo context.
• docs/spec-guide.md is the practical authoring guide for HarnessSpec.
• docs/design.md is the internal architecture document: compiler pipeline, kernel, validator wall, eval gauntlet, climber, and freeze artifact.
• docs/tasks.md is the phased build plan and acceptance criteria.
• docs/build-log.md is the chronological implementation record.
• docs/lineage.md explains the earlier experiments and the constraints inherited from them.

Evidence reports:

• reports/implementation-report.md covers Phases 0-3.
• reports/phases-4-5-report.md covers the full gauntlet, simulator, economics, and hill-climber.
• reports/eva-expansion-report.md covers the EVA expansion and current residuals.

• compiler/: packet loading, profiling, synthesis orchestration, and artifact handling.
• spec/: HarnessSpec schema and loading.
• kernel/: policy-free runtime for executing specs and judgment slots.
• wall/: static validator wall and known-bad corpus.
• gauntlet/: split evals, genericity, simulator, sandbox, and baseline scoring.
• climber/: trace-driven mutation loop with regression gates.
• packets/: task evidence packets.
• harnesses/ and harnesses_generated/: hand-authored and generated harness artifacts used by the reports.

Phases 0-5, the tech-debt sweep, and the first EVA expansion are implemented. The strongest remaining research gap is coordinated multi-file mutation in the climber: the loop can find real holes, preserve them as eval surface, and fix single-artifact clusters, but some residuals require paired spec+prompt edits.

Code is Apache-2.0 - see LICENSE. Some task packets include benchmark-derived or scenario-derived fixtures with their own provenance notes; check each packet's task.yaml before reusing packet data outside this repo.