█████╗ ██╗      ██████╗ ██╗  ██╗
██╔══██╗██║      ██╔══██╗██║ ██╔╝
███████║██║█████╗██████╔╝█████╔╝ 
██╔══██║██║╚════╝██╔═══╝ ██╔═██╗ 
██║  ██║██║      ██║     ██║  ██╗
╚═╝  ╚═╝╚═╝      ╚═╝     ╚═╝  ╚═╝

Real-world AI Coding Agent Performance Benchmark

🇨🇳 中文版 | English

AI-PK is an objective, reproducible benchmark system for AI coding assistants. It evaluates different AI engines and IDE clients through real-world Zig project development tasks (ZigScan port scanner).

Test Task: Build a complete Zig port scanner from scratch, including concurrent scanning, argument parsing, performance optimization, and other real development requirements.

📊 View Full Leaderboard | Download Text Report

• Total Tests: 22
• ✅ Success: 9 (41%)
• ⚠️ Partial Success: 6 (27%)
• ❌ Failed: 7 (32%)

• 🥇 Highest Quality: Claude Sonnet 4.5 + Factory Droid (10/10 Perfect Score)
• ⚡ Fastest: Claude Sonnet 4.5 + Factory Droid (15 minutes)
• 💰 Most Token Efficient: Claude Sonnet 4.5 + Factory Droid (172.5K tokens)

International:

• Claude Sonnet 4.5, Claude Opus 4.1 (Anthropic)
• GPT-5 with multiple configuration levels (OpenAI)
• Grok (xAI)
• Supernova

Chinese:

• Qwen (Alibaba)
• GLM-4.6 (Zhipu AI)
• Kat (Kuaishou)

• Factory Droid
• Codex CLI
• Roo Code
• Kilo Code
• ClaudeCode
• Cline
• Qwen-CLI

• 🌐 English Report - Sortable tables with embedded charts
• 🌐 Chinese Report - 中文版报告

• 📄 English Text Report
• 📄 Chinese Text Report

• 📊 JSON Data - Complete structured data

• 📈 Success Rate Distribution
• 📈 Token Efficiency Analysis
• 📈 Engine Comparison
• 📈 Quality Heatmap

Uses a standardized 0-10 scoring system. See Quality Scoring Standards for details.

Scoring Formula: Final Score = min(max(Base + Bonus - Penalty, 0), 10)

• Base Score:

  • SUCCESS: 8 points
  • PARTIAL: 5 points
  • FAILED: 0 points

• Bonus (up to +3):

  • Functionality completeness (0-1)
  • Code quality (0-1)
  • Performance (0-1)

• Penalty (up to -5):

  • Bug severity (0-2)
  • Manual intervention needed (0-2)
  • Efficiency issues (0-1)

# Open interactive reports in browser
xdg-open results/REPORT_EN.html  # English version
xdg-open results/REPORT_ZH.html  # Chinese version

# Or view text reports
cat results/BENCHMARK_REPORT.txt

# 1. Navigate to project directory
cd ai-pk

# 2. Run full analysis
bash scripts/run_all.sh

# Generated files:
# - results/BENCHMARK_REPORT.txt (English)
# - results/BENCHMARK_REPORT_ZH.txt (Chinese)
# - results/REPORT_EN.html (Interactive English)
# - results/REPORT_ZH.html (Interactive Chinese)
# - results/charts/*.png (Visualization charts)

ai-pk/
├── benchmarks/
│   └── zigscan/              # ZigScan test results (22 tests)
│       ├── sonnet4.5-dorid-2025-10-25/
│       │   ├── stats.json    # Standardized data
│       │   ├── finish.log    # Test log
│       │   └── src/          # Generated code
│       ├── gpt5_hight-dorid/
│       └── ...
├── scripts/
│   ├── cyberpunk_analyzer.py      # Main analysis script
│   ├── generate_charts.py         # Chart generator
│   ├── generate_bilingual_html.py # Bilingual HTML reports
│   └── run_all.sh                 # One-click run
├── results/
│   ├── BENCHMARK_REPORT.txt       # English text report
│   ├── BENCHMARK_REPORT_ZH.txt    # Chinese text report
│   ├── REPORT_EN.html             # English interactive report
│   ├── REPORT_ZH.html             # Chinese interactive report
│   ├── benchmark_data.json        # Complete data
│   └── charts/                    # Visualization charts
├── QUALITY_SCORING_STANDARD.md    # Scoring standards
└── README.md                      # This file

Each test includes a stats.json file with the following structure:

{
  "test_dir": "sonnet4.5-dorid-2025-10-25",
  "engine": "Claude Sonnet 4.5",
  "client": "Factory Droid",
  "completed": "SUCCESS",
  "time_minutes": 15,
  "tokens": 172500,
  "quality_score": 10,
  "quality_breakdown": {
    "base_score": 8,
    "bonus": { "functionality": 1.0, "code_quality": 1.0, "performance": 1.0 },
    "penalty": { "bugs": 0.0, "workaround": 0.0, "efficiency": 0.0 }
  }
}

Contributions of new test cases are welcome! See CONTRIBUTING.md

1. Create a new directory in benchmarks/zigscan/
2. Add a stats.json file (refer to existing format)
3. Run bash scripts/run_all.sh to regenerate reports
4. Submit a PR

See RELEASE_NOTES.md for the latest updates.