GBQA: Towards Industrial-Level Quality Assurance Evaluation for Agents

Automated bug discovery in real-world software environments

An open-source benchmark framework for running agents against real GitHub software releases, letting agents explore the live environment, discover latent bugs, and receive verifier-backed QA scores.

The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for LLMs. A GBQA instance points to a real GitHub software release, gives one targeted bug hint, defines how that software should run in an isolated sandbox, exposes supported interaction modes, and evaluates one issue report with rule-based function-level validation.

GBQA requires Python 3.12 or newer.

pip install -e .

Create a root .env file from the template:

cp .env.example .env

Fill in the required runtime fields:

DAYTONA_API_KEY=
# Required only for `-e modal` when ~/.modal.toml is not configured.
MODAL_TOKEN_ID=
MODAL_TOKEN_SECRET=
API_KEY=
BASE_URL=https://zenmux.ai/api/v1
MODEL_NAME=
GITHUB_TOKEN=

Run the default GBQA Harbor agent against a task package in a remote Daytona sandbox:

python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/<task-id> \
  -e daytona \
  --gbqa-task-runner gbqa \
  --ak interaction_mode=terminal \
  --ak max_steps=10

Modal can be used as the Harbor sandbox provider with the same GBQA task package and agent harness. Authenticate Modal first with modal token new or set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET in .env, then switch only the environment flag:

python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/<task-id> \
  -e modal \
  --gbqa-task-runner gbqa \
  --ak interaction_mode=terminal \
  --ak max_steps=10

pip install -e . installs Modal's API proxy support. This matters on machines with HTTP_PROXY / HTTPS_PROXY or similar proxy variables set; without it the Modal SDK can fail before sandbox creation with a missing python-socks dependency.

Use --ak max_steps=10 as a fast infrastructure smoke. Use a larger budget such as --ak max_steps=50 when checking whether the agent can reproduce the hinted target bug and produce a function-level pinpoint. The default minimal harness includes source-code tools because targeted GBQA rewards require pinpoint evidence. Use --ak harness_mode=full only when the run should also exercise logs, automatic diagnostics, and worker subagents.

Use browser interaction by switching the interaction mode:

python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/<task-id> \
  -e daytona \
  --gbqa-task-runner gbqa \
  --ak interaction_mode=browser \
  --ak max_steps=10

GBQA supports two task-running paths:

• GBQAHarborAgent, the custom QA harness. It uses provider-neutral API_KEY, BASE_URL, and MODEL_NAME.
• Harbor built-in CLI agents such as codex and claude-code. These can use Codex / Claude Code subscription auth.

Use the GBQA launcher selectors to choose the path:

# Custom GBQA QA harness
python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/dark-castle-key-fragment-combine \
  -e daytona \
  --gbqa-task-runner gbqa \
  --ak interaction_mode=terminal \
  --ak max_steps=10

# Harbor built-in Claude Code with subscription auth
claude setup-token
export CLAUDE_CODE_OAUTH_TOKEN="claude_oauth_..."
python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/dark-castle-key-fragment-combine \
  -e daytona \
  --gbqa-task-runner claude-code \
  --gbqa-agent-model anthropic/claude-sonnet-4-6 \
  --gbqa-agent-auth subscription

# Harbor built-in Codex with subscription auth
codex login
python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks/dark-castle-key-fragment-combine \
  -e daytona \
  --gbqa-task-runner codex \
  --gbqa-agent-model gpt-5 \
  --gbqa-agent-auth subscription \
  --gbqa-codex-auth-file "$HOME/.codex/auth.json"

The task instruction tells generic CLI agents to start Dark Castle and write /logs/agent/gbqa/issue.json, which is the verifier's preferred input. The GBQA harness writes the same artifact through its final issue-report pass before exit. A single-element legacy bugs.json is still accepted for compatibility. The verifier runtime is bundled under each task's tests/_gbqa_runtime, so generic CLI agents do not need to upload /sandbox/gbqa.

See docs/subscription-auth.md for the Harbor task-runner subscription authentication reference.

GBQA's gbqa.cli.harbor_run wrapper loads the root .env and forwards all arguments to Harbor. When a local path or registered dataset contains many task packages, Harbor can launch multiple remote sandboxes at the same time and run one evaluation per task environment. Daytona and Modal are selected with the native Harbor -e daytona or -e modal flag.

For example, once gbqa/tasks contains many verified task packages, run up to 100 task evaluations concurrently:

python -m gbqa.cli.harbor_run run \
  -p gbqa/tasks \
  -e daytona \
  --gbqa-task-runner gbqa \
  --ak interaction_mode=terminal \
  --ak max_steps=10 \
  --n-tasks 100 \
  --n-concurrent 100

Here --n-concurrent controls how many Harbor trials can run at once. With Daytona or Modal, that means many independent remote sandboxes can be active in parallel. It is not intended to create multiple concurrent agents inside the same task sandbox.

For local Linux verification, run:

bash scripts/test-linux.sh

Set GBQA_RUN_AGENT_SCRIPT_SMOKES=1 to also run the standalone agent/test/test_*.py smoke scripts. Set GBQA_RUN_NETWORK_SMOKES=1 only when you intentionally want to include model/API network smoke tests.

In Harbor benchmark runs, evaluation is performed automatically by the verifier phase after the agent writes normalized artifacts. The agent/ harness does not read target-bug verifier assets or compute benchmark scores.

GBQA's default verifier reward is targeted and binary. Each benchmark instance has one known target bug and one selected agent-facing hint; official instance data can store weak, medium, and strong hint variants for calibration. The verifier reads the submitted issue report, checks required issue fields, and gives reward 1.0 only when report_status=complete and the reported pinpoint aligns with the target golden patch. Pinpoint can be a source location or a SWE-style minimal patch/diff hunk.

• Agent artifacts: /logs/agent/gbqa/run.json, /logs/agent/gbqa/issue.json, /logs/agent/gbqa/bugs.json, /logs/agent/gbqa/steps.jsonl
• Each issue.json includes top-level report_status, exit_status, and missing_fields; the nested issue should include expected_behavior, observed_fault, reproduction, and source-level pinpoint via locations[] or patch/diff
• Harbor reward outputs: /logs/verifier/reward.txt, /logs/verifier/reward.json
• Full GBQA evaluation payload: /logs/verifier/gbqa_result.json

Each benchmark instance is a Harbor-compatible package under gbqa/tasks/<instance-id>. Multiple instances may share the same upstream software release, but each instance has its own instruction.md, gbqa.yaml, target bug file, verifier entrypoint, and artifact contract. Running multiple instances launches multiple independent Harbor trials and remote sandboxes.

Environment discovery and preparation live outside the runtime package in environment/. This offline toolchain searches GitHub repositories, detects deployable sub-environments, filters and ranks candidates, runs optional Daytona deployment verification, supports human review, and exports approved task packages into gbqa/tasks.

python -m environment.sourcing.cli run \
  --provider github \
  --query "archived:false fork:false stars:>=10 mirror:false" \
  --limit 500 \
  --top-k 100 \
  --output-dir environment/catalog/runs/dev

python -m environment.export.cli generate \
  --input environment/catalog/runs/dev/approved_task_seeds.jsonl \
  --output gbqa/tasks

• GBQAHarborAgent as the default custom QA agent wrapper.
• Example real GitHub software environment: Dark Castle in a remote Daytona sandbox.
• Terminal and browser interaction modes for the example task.
• Harbor-compatible verifier and reward outputs.

• Keep Harbor built-in codex and claude-code task runners selectable through gbqa.cli.harbor_run, including subscription auth.
• Add optional custom QA-harness wrappers such as CodexHarborAgent and ClaudeCodeHarborAgent if they provide GBQA-specific behavior beyond Harbor's built-in CLI agents.
• Support local sandbox + colocated agent.
• Support local agent + remote sandbox.
• Add more verified benchmark environments and task manifests.
• Keep Daytona and Modal selectable as Harbor-managed remote sandbox providers.
• Scale parallel evaluation across remote sandboxes.

• Support terminal, browser, computer, and mixed interaction methods, with task metadata recording concrete terminal surfaces such as API, CLI, or Python API.
• Extend sandbox support from Linux toward Windows and macOS.
• Run broader LLM evaluation experiments and release a leaderboard.

• Collect trajectory data.
• Standardize reward signals.
• Support RL training and optimization workflows.

Contributions are welcome. The highest-priority areas are new Harbor-compatible task packages, additional agent harness adapters, verifier improvements, and sandbox/runtime robustness.