Skip to content
keyanUBPublic

About

Python MCP server for OWASP secure coding policy selection, with coding-agent integrations and comparative evaluations

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

PECA

A security-focused harness for coding agents, with reproducible experiments.

PECA studies whether secure-coding guidance and independent public verification improve generated code. It measures functionality, benchmark security outcomes, agent completion and actual cost separately. SecRepoBench is an evaluation dataset—not a source of task-specific harness rules or an eligibility filter.

The current repository workflow uses the OpenHands SDK through an isolated shell adapter. It is not the stock interactive OpenHands CLI.

Architecture · Run an experiment · Runbook · Documentation

Harness architecture

flowchart LR
    TASK["Public task<br/>and code context"] --> AGENT["Coding Agent"]

    subgraph HARNESS["Security Harness"]
        POLICY["Select relevant<br/>security policies"]
        VERIFY["Independently verify<br/>candidate code"]
        CONTROL["Control<br/>scope-limited repair"]
        VERIFY --> CONTROL
    end

    TASK --> POLICY
    POLICY -->|"Security guidance"| AGENT
    AGENT --> CODE["Candidate code"]
    CODE --> VERIFY
    CONTROL -->|"Failure feedback · target-only repair"| AGENT
    CONTROL -->|"Stop"| OUTPUT["Code and<br/>verification status"]
Loading

The full configuration has two intervention points: context-relevant guidance before generation, and independent verification feedback after generation. The four conditions below enable or disable guidance and external repair.

Verification uses public build/tests with requested ASan/UBSan instrumentation; coverage is not guaranteed. Repairs can modify only generated submission targets, not dependencies or tests. The controller may stop when checks pass, repair is not eligible, or configured limits are reached. Output is code with a verification status, not a guarantee of security.

Final benchmark evaluation is outside this mechanism and has no feedback path to the agent or Advisor. CLI stages, sealing and cost reporting are described below and in the runbook, rather than in the mechanism diagram.

The diagram is editable Mermaid source. Keep it aligned with the implementation; see the component map and update checklist.

Run an experiment

Prerequisites

  • Linux or Ubuntu under WSL, Python 3.12+ and a working Docker daemon.
  • A host environment with the Advisor package installed; the examples use .venv/bin/python.
  • A separate Python environment with the compatible OpenHands SDK, supplied with --openhands-python.
  • The pinned SecRepoBench checkout and required agent/ARVO images already present. The experiment scripts do not pull images automatically.
  • OPENAI_API_KEY set securely in the host environment for generation only. Do not put credentials in command-line arguments or committed files.

See the runbook for setup details. Run the following commands from the repository root. Replace TASK_ID with a task chosen before observing results and use a new experiment directory.

1. Generate and seal

This stage makes real model calls. It freezes the configuration, runs coding and public-only verification/repair, then seals all planned slots. It does not perform final benchmark evaluation.

.venv/bin/python scripts/run_experiment.py \
  --source .artifacts/sources/SecRepoBench \
  --output .artifacts/NEW-EXPERIMENT \
  --tasks TASK_ID \
  --conditions baseline policy verification full \
  --openhands-python /absolute/path/to/openhands-env/bin/python

2. Evaluate sealed candidates

Run only after generation completes and writes seal.json. This stage checks frozen integrity and runs final benchmark tests, without model calls or repairs.

.venv/bin/python scripts/evaluate_experiment.py \
  --source .artifacts/sources/SecRepoBench \
  --experiment .artifacts/NEW-EXPERIMENT

3. Summarize results and efficiency

Run after final evaluation completes. This writes summary.json and summary.md without generating code or rerunning tests.

.venv/bin/python scripts/summarize_experiment.py \
  --experiment .artifacts/NEW-EXPERIMENT

Omitting --tasks declares all official tasks and can incur substantial cost. Do not run stages concurrently or alter the frozen implementation between generation and evaluation. Exit 0 means a stage completed its records—not that every agent or security check succeeded.

See the runbook for parameters, outputs, exit codes, partial reports and interruption handling. To test existing code without models, use the separate testbed CLI, not the formal experiment workflow.

Four experiment conditions

Condition ID Security guidance External repair using public verification
baseline None No
policy — Advisor-only Advisor-selected policies No
verification None Yes, bounded
full Advisor-selected policies Yes, bounded

Independent public checks are recorded for every candidate; only the repair conditions feed failures back. Agents may run their own public tests in every condition. Repair permissions cover only the generated submission target, not arbitrary dependency or test files. Budget and scope details.

Fairness and interpretation

  • Declare the population before outcomes; retain failed and unavailable slots. Do not filter tasks through qualification or add task-specific test fixes.
  • Keep benchmark references, PoCs and final results out of generation and repair. All planned generations end before final evaluation begins.
  • Preserve every attempt. No selective reruns, overwriting results or resuming generation after sealing.
  • Report actual cost and verification overhead; do not force equal expenditure. Missing or checkpoint-only usage is unknown/incomplete, not zero.
  • A benchmark pass is not proof of general security. Previously inspected tasks are not untouched held-out data; model pretraining contamination is not ruled out.

Development and current scope

The four-condition repository workflow, generated-file-only repair permissions and separate evaluation/reporting CLIs are implemented. Sanitizer coverage and the connection between selected policies and executable checks remain incomplete. All-SCP and SCP-RAG are planned ablations, not implemented conditions.

Run software checks without model calls or benchmark evaluation:

.venv/bin/python -m pytest harness/tests policy-advisor-mcp/tests -q
# Include Docker-based synthetic fixtures:
PECA_TEST_DOCKER=1 .venv/bin/python -m pytest harness/tests policy-advisor-mcp/tests -q

Maintain the diagram and documentation with the implementation. Keep experiment evidence under .artifacts/ (Git-ignored), including failed attempts; do not reuse retired exploratory results as evidence of effectiveness.

Documentation

Need Start here
Run or troubleshoot an experiment Experiment runbook
Understand isolation, repairs and the architecture Repository harness
Understand scoring and benchmark limitations Benchmark evaluation
Build/test existing code without models SecRepoBench testbed
Install or use the independent Policy Advisor Advisor MCP
Launch interactive OpenHands outside formal experiments Interactive launcher
Review research design and planned comparisons Design protocol · Ablation plan
Explore other verifier tasks or optional analysis Trusted verifier · AST context
Check integration status or migrate older setups Client integrations · Migration

The OWASP policy data retains its upstream attribution and license: data notice.

About

Python MCP server for OWASP secure coding policy selection, with coding-agent integrations and comparative evaluations

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages