Kode Agent Eval Harness

Evaluate KODE on SWE, TB2, and Tau through a standalone benchmark adapter layer, without coupling benchmark logic to the SDK repository.

• Runs swebench-verified through Harbor
• Runs terminal-bench@2.0 through Harbor
• Runs official tau-bench
• Uses the published npm package @shareai-lab/kode-sdk
• Pulls benchmark frameworks dynamically at runtime
• Supports GitHub Actions smoke tests and larger benchmark runs
• Produces merged summaries and per-task result artifacts

Install dependencies:

npm install

Run local validation:

npm run typecheck
npm run bench:bundle:harbor
npm run bench:bundle:tau

Model names use provider/model format.

Example:

glm/glm-5

            +----------------------+
            |  GitHub Actions      |
            |  SWE / TB2 / Tau     |
            +----------+-----------+
                       |
          +------------+-------------+
          |                          |
          v                          v
  +---------------+          +---------------+
  | Harbor Path   |          | Tau Path      |
  | SWE / TB2     |          | Tau           |
  +-------+-------+          +-------+-------+
          |                          |
          v                          v
  +---------------+          +---------------+
  | Harbor Adapter|          | Tau Adapter   |
  +-------+-------+          +-------+-------+
          |                          |
          v                          v
  +---------------+          +---------------+
  | Node Runner   |          | Step Runner   |
  +-------+-------+          +-------+-------+
          \____________________  ____________/
                               \/
                    +----------------------+
                    | @shareai-lab/kode-sdk|
                    +----------------------+

Use the manual dispatch workflows:

• .github/workflows/eval-swe.yml
• .github/workflows/eval-tb2.yml
• .github/workflows/eval-tau.yml

Minimum required input:

model_name=provider/model

Typical repository configuration:

• OPENAI_API_KEY
• OPENAI_BASE_URL
• OPENAI_API
• ANTHROPIC_API_KEY
• GEMINI_API_KEY
• MINIMAX_API_KEY

For OpenAI-compatible GLM routing, the harness reads:

• OPENAI_API_KEY
• OPENAI_BASE_URL

Common workflow inputs:

• model_name
• task_names
• task_limit
• shard_count
• max_parallel_shards
• n_attempts
• n_concurrent

• model_name
• tau_env
• task_split
• task_ids
• start_index
• end_index
• num_trials
• max_concurrency

Each run produces:

• shard artifacts
• per-task result.json
• agent/kode-result.json
• merged summary markdown
• merged results JSON
• merged per-test details JSON

Each run produces:

• final metrics JSON
• per-task reward data
• per-task trajectory data

See:

• README.zh-CN.md