👨‍💻 Awesome Code Benchmark

A comprehensive code domain benchmark review of LLM researches.

• Taxonomy
• Surveys
• Benchmark Categories
  • Repository & Agentic Software Engineering
  • Program Repair, Testing & Debugging
  • Security, Reliability & Robustness
  • Code Understanding, Search & Review
  • Performance Optimization
  • Frontend, UI & Visual-Interactive Development
  • Code Generation & Completion

This list organizes code benchmarks by primary capability and software-engineering workflow. Each benchmark entry includes compact metadata for task type, granularity, interaction pattern, and evaluation method.

1. Software Development Life Cycle Perspective A Survey of Benchmarks for Code Large Language Models and Agents from Xi’an Jiaotong University

2. Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks from Zhejiang University

3. A Survey on Large Language Model Benchmarks from Shenzhen Key Laboratory for High Performance Data Mining

• ProjDevBench (2026): ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development - task: End-to-end project development - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / LLM-as-Judge - Github
• HWE-Bench (2026): HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - task: Hardware repository bug repair - granularity: Repository - interaction: Agentic - evaluation: Execution
• SpecBench (2026): Measuring Reward Hacking in Long-Horizon Coding Agents - task: Reward hacking / spec compliance - granularity: Systems-level task (JSON parser → OS kernel) - interaction: Agentic - evaluation: Held-out vs visible tests - Github
• IDE-Bench (2026): IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks - task: IDE-native software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
• RepoGenesis (2026): RepoGenesis: Benchmarking End-to-End Microservice Generation from Readme to Repository - task: End-to-end microservice repository generation - granularity: Project / Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Deployment - Github
• SWE-Refactor (2026): SWE-Refactor: A Repository-Level Benchmark for Real-World LLM-Based Code Refactoring - task: Repository-level refactoring - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Compilation / Unit Tests / Static Analysis
• SWE-ContextBench (2026): SWE Context Bench: A Benchmark for Context Learning in Coding - task: Context reuse across related coding tasks - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Cost
• SWE-Bench Mobile (2026): SWE-Bench Mobile: Can Large Language Model Agents Develop Industry-Level Mobile Applications? - task: Industrial mobile application development - granularity: Repository / Workflow - interaction: Multimodal / Agentic - evaluation: Unit Tests / Execution - 🌐Website
• RepoMod-Bench (2026): RepoMod-Bench: A Benchmark for Code Repository Modernization via Implementation-Agnostic Testing - task: Repository modernization / migration - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution - Github
• SWE-QA-Pro (2026): SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - task: Agentic repository-level code understanding - granularity: Repository - interaction: Agentic - evaluation: QA Accuracy / Execution
• SWE-STEPS (2026): Beyond Isolated Tasks: A Framework for Evaluating Coding Agents on Sequential Software Evolution - task: Sequential software evolution - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Execution / Static Analysis
• RepoZero (2026): RepoZero: Can LLMs Generate a Code Repository from Scratch? - task: Repository generation from scratch - granularity: Project / Repository - interaction: Agentic - evaluation: Black-box Tests / Execution
• VISTA (2026): VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents - task: Visual spec-to-web-app development - granularity: Project / Workflow - interaction: Multimodal / Agentic - evaluation: Browser Tests / Visual Similarity / DOM Matching - Github 🌐Website 🤗Dataset
• RAMP (2026): Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems - task: Production runtime assessment - granularity: Project / Workflow - interaction: Agentic - evaluation: Execution / Runtime / Cost - 🌐Website
• Claw-SWE-Bench (2026): Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks - task: Agent harness evaluation for issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution / Cost - Github 🤗Dataset
• Dialogue SWE-Bench (2026): Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents - task: Dialogue-driven issue resolving - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Unit Tests / Dialogue Quality
• SWE-Future (2026): SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents - task: Future-oriented repository task synthesis - granularity: Repository - interaction: Agentic - evaluation: Semantic Matching / Execution
• SWE-Lancer (2025): SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering? - task: Freelance software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution / Human / Economic - Github
• Multi-SWE-bench (2025): Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving - task: Multilingual issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• SWE-Bench Pro (2025): SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? - task: Long-horizon issue resolving - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution
• PRDBench (2025): Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation - task: PRD-driven project development - granularity: Project / Repository - interaction: Agentic - evaluation: Execution / LLM-as-Judge
• LongCodeBench (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Long-context code comprehension and repair - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Unit Tests / Execution / LLM-as-Judge - 🤗Dataset
• SWE-Bench++ (2025): SWE-Bench++: A Framework for the Scalable Generation of Software Engineering Benchmarks from Open-Source Repositories - task: Repository-level bug fixing and feature requests - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• AInsteinBench (2025): AInsteinBench: Benchmarking Coding Agents on Scientific Repositories - task: Scientific repository development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
• SWE-PolyBench (2025): SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents - task: Repository-level issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website 🤗Dataset
• CoreCodeBench (2025): CoreCodeBench: A Configurable Multi-Scenario Repository-Level Benchmark - task: Repository-level software engineering - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution - Github 🤗Dataset
• GitTaskBench (2025): GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging - task: Repository-level coding tasks - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution - Github 🌐Website
• CodeAssistBench (2025): CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance - task: Multi-turn code assistance - granularity: Function / Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Human / LLM-as-Judge
• SWE-Dev (2025): SWE-Dev: Evaluating and Training Autonomous Feature-Driven Software Development - task: Feature-driven software development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• SWE-Flow-Eval (2025): SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner - task: Test-driven incremental development - granularity: Repository / Workflow - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• LoCoBench-Agent (2025): LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering - task: Long-context software engineering workflows - granularity: Repository / Workflow - interaction: Multi-turn / Agentic - evaluation: Execution / Tool-use Metrics / LLM-as-Judge

• RepoExploreBench (2026): Planning to Explore: Curiosity-Driven Planning for LLM Test Generation - task: Iterative test generation - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• CI-Repair-Bench (2026): CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows - task: CI failure diagnosis and repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: CI / Execution
• ArkEval (2026): ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS - task: ArkTS automated program repair - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Unit Tests / Execution
• HEJ-Robust (2026): HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair - task: Robust automated program repair - granularity: Function - interaction: Single-turn - evaluation: Unit Tests / Execution
• FixtureEval (2026): Fixturize: Bridging the Fixture Gap in Test Generation - task: Test fixture classification and generation - granularity: Function / File - interaction: Agentic - evaluation: Unit Tests / Execution
• RESTestBench (2026): RESTestBench: A Benchmark for Evaluating the Effectiveness of LLM-Generated REST API Test Cases from NL Requirements - task: Requirement-based REST API test generation - granularity: API / Workflow - interaction: Single-turn / Multi-turn - evaluation: Mutation Testing / Execution
• QBugLM (2026): QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging - task: Quantum software debugging and repair - granularity: File / Project - interaction: Multi-agent - evaluation: Simulation / Execution
• A11YBench (2026): A11YRepair: Bridging Web Accessibility Barriers via Knowledge-Enhanced Divide-and-Conquer Repair - task: Web accessibility repair - granularity: Project / Repository - interaction: Agentic - evaluation: Browser Tests / Execution
• AgentDefect (2026): SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agents - task: LLM-agent bug repair - granularity: File / Project - interaction: Agentic - evaluation: Unit Tests / Execution
• HWE-Bench (2026): HWE-Bench: Benchmarking LLM Agents on Real-World Hardware Bug Repair Tasks - task: Hardware repository bug repair - granularity: Repository - interaction: Agentic - evaluation: Simulation / Execution
• LiveCodeBench (2025): LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
• COAST (2025): COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
• SWE-bench Multimodal (2025): SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
• FeedbackEval (2025): FeedbackEval A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• CVE-Bench (2025): CVE-Bench:Benchmarking LLM-based Software Engineering Agent’s Ability to Repair Real-World CVE Vulnerabilities - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github Dataset
• OmniGIRL (2025): OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue Resolution - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
• LongSWE-Bench (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - 🤗Dataset
• VADER (2025): VADER: A Human-Evaluated Benchmark for Vulnerability Assessment, Detection, Explanation, and Remediation - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• Breakpoint (2025): Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• MLDebugging (2025): MLDebugging: Towards Benchmarking Code Debugging Across Multi-Library Scenarios - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• Skywork-SWE (2025): Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• SWE-MERA (2025): SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
• BuildBench (2025): BuildBench: Benchmarking LLM Agents on Compiling Real-World Open-Source Software - task: Build and compilation repair - granularity: Repository / Workflow - interaction: Agentic - evaluation: Execution
• SWT-Bench (2024): SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website
• HumanEvalPack (2024): OctoPack: Instruction Tuning Code Large Language Models - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
• SWE-bench (2024): SWE-bench: Can Language Models Resolve Real-World GitHub Issues? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🌐Website
• GitBug-Java (2024): GitBug-Java: A Reproducible Benchmark of Recent Java Bugs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website
• GitBug-Actions (2024): GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github ▶️Video
• RepoBugs (2024): When Large Language Models Confront Repository-Level Automatic Program Repair: How Well They Done? - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• RepoFixEval (2024): RepoFixEval: A Repository-Level Program Repair Benchmark From Issue Discovering to Bug Fixing - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Link
• DebugBench (2024): DebugBench: Evaluating Debugging Capability of Large Language Models - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github 🤗Dataset
• Multi-Bug (2024): Instruct, Not Assist: LLM-based Multi-Turn Planning and Hierarchical Questioning for Socratic Code Debugging - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• Coffee-Gym (2024): Coffee-Gym: An Environment for Evaluating and Improving Natural Language Feedback on Erroneous Code - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - 🤗Dataset
• INTERVENOR (2024): INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github
• StatType-SO (2024): ZS4C: Zero-Shot Synthesis of Compilable Code for Incomplete Code Snippets using LLMs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution
• Buggy-HumanEval&Buggy-FixEval (2023): Large Language Models of Code Fail at Completing Code with Potential Bugs - task: Bug fixing / issue resolving - granularity: Repository - interaction: Agentic - evaluation: Unit Tests / Execution - Github Dataset

• RealSec-bench (2026): RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories - task: Secure code generation - granularity: Repository - interaction: Single-turn / Agentic - evaluation: SecurePass / SAST / Execution
• HardSecBench (2026): HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generation - task: Hardware and firmware secure code generation - granularity: Function / Project - interaction: Single-turn / Agentic - evaluation: Unit Tests / Security Exploit / Execution
• TOSSS (2026): TOSSS: a CVE-based Software Security Benchmark for Large Language Models - task: Secure-vs-vulnerable code selection - granularity: Function / Snippet - interaction: Single-turn - evaluation: Security Score
• Multi-LLMSecCodeEval (2026): Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval - task: Secure code generation and remediation pipeline evaluation - granularity: Function / Workflow - interaction: Multi-agent - evaluation: SAST / Security Exploit / Execution
• RealVuln (2026): RealVuln: Benchmarking Rule-Based, General-Purpose LLM, and Security-Specialized Scanners on Real-World Code - task: Vulnerability detection in real-world code - granularity: Repository - interaction: Single-turn / Agentic - evaluation: Vulnerability Detection / F-score - 🌐Website
• Delulu (2026): Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks - task: Code hallucination detection - granularity: Function / File - interaction: Single-turn - evaluation: Compilation / Execution / Human - Github
• SEC-bench Pro (2026): SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks? - task: Long-horizon software security bug hunting - granularity: Repository / Workflow - interaction: Agentic - evaluation: PoC / Oracle / Execution
• RewardHackingAgents (2026): RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agents - task: Agent evaluation-integrity robustness - granularity: Project / Workflow - interaction: Agentic - evaluation: Integrity / Runtime / Execution
• CodeHalu (2025): CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github 🤗Dataset
• APIHulBench (2025): Towards Mitigating API Hallucination in Code Generated by LLMs with Hierarchical Dependency Aware - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github
• THINK (2025): THINK: Tackling API Hallucinations in LLMs via Injecting Knowledge - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - Github 🤗Dataset
• aiXamine (2025): aiXamine: Simplified LLM Safety and Security - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - 🌐Website
• SafeGenBench (2025): SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
• CodeMirage (2025): CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
• SEC-bench (2025): SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset📊LeaderBoard
• RAS-Eval (2025): RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
• JsDeObsBench (2025): JsDeObsBench: Measuring and Benchmarking LLMs for JavaScript Deobfuscation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 📊Leaderboard
• CIRCLE (2025): Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - 🤗Dataset
• MOCHA (2025): MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts? - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
• A.S.E (2025): A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution
• HALLUCODE (2024): Exploring and Evaluating Hallucinations in LLM-Powered Code Generation - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution
• Collu-Bench (2024): Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code - task: Reliability / hallucination / robustness - granularity: Function / API - interaction: Single-turn - evaluation: Execution - 🤗Dataset
• RedCode (2024): RedCode: Risky Code Execution and Generation Benchmark for Code Agents - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🌐Website 📊LeaderBoard
• CodeWMBench (2024): CodeWMBench: An Automated Benchmark for Code Watermarking Evaluation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
• RMCBench (2024): RMCBench: Benchmarking Large Language Models' Resistance to Malicious Code - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
• PyP4LLMSec (2024): Benchmarking the Security Aspect of Large Language Model-Based Code Generation - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset
• CWE-Bench-Java (2024): IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
• CyberSecEval 3 (2024): CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset
• CS-Eval (2024): CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github 🤗Dataset
• SecBench (2024): SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Dataset 🌐Website
• COCO (2023): COCO: Testing Code Generation Systems via Concretized Instructions - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github
• ReCode (2023): ReCode: Robustness Evaluation of Code Generation Models - task: Security / vulnerability handling - granularity: Repository - interaction: Agentic - evaluation: Security Exploit / Execution - Github Dataset

• SWE-PRBench (2026): SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback - task: Pull request review - granularity: Repository / Workflow - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
• SWE-QA-Pro (2026): SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding - task: Repository-level code understanding - granularity: Repository - interaction: Agentic - evaluation: QA Accuracy / Execution
• Code-QA-Bench (2026): Code-QA-Bench: Separating Code Reasoning from Documentation Memorization in Repository-Level QA - task: Repository-level code QA - granularity: Repository - interaction: Agentic - evaluation: LLM-as-Judge / Human
• CORE-Bench (2026): CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding - task: Repository-aware code retrieval - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Retrieval Metrics
• CoREB (2026): Beyond Retrieval: A Multitask Benchmark and Model for Code Search - task: Code retrieval and reranking - granularity: Function / Repository - interaction: Single-turn - evaluation: nDCG / Retrieval Metrics
• CLARC (2026): CLARC: C/C++ Benchmark for Robust Code Search - task: Robust code search - granularity: Function / Repository - interaction: Single-turn - evaluation: Retrieval Metrics - 🤗Dataset
• R2Eval (2026): Evaluating LLMs Code Reasoning Under Real-World Context - task: Real-world code reasoning - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / Accuracy
• CodeRQ-Bench (2026): Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks - task: Code reasoning quality evaluation - granularity: Function / File - interaction: Single-turn - evaluation: Human / LLM-as-Judge - Github
• CodeGlance (2026): CodeGlance: Understanding Code Reasoning Challenges in LLMs through Multi-Dimensional Feature Analysis - task: Code behavior reasoning - granularity: Function / API - interaction: Single-turn - evaluation: Accuracy / Feature Analysis
• code-logic-bench (2026): Imandra CodeLogician: Neuro-Symbolic Reasoning for Precise Analysis of Software Logic - task: Formal software logic reasoning - granularity: Function / Program - interaction: Agentic - evaluation: Formal Verification / Accuracy
• CodeJudge-Eval (2025): CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
• CodeMMLU (2025): CodeMMLU: A Multi-Task Benchmark for Assessing Code Understanding Capabilities of CodeLLMs - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset 🌐Website 📊LeaderBoard
• LongCodeQA (2025): LongCodeBench: Evaluating Coding LLMs at 1M Context Windows - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - 🤗Dataset
• CTF-Code (2025): Success is in the Details: Evaluate and Enhance Details Sensitivity of Code - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
• CodeSense (2025): CodeSense: a Real-World Benchmark and Dataset for Code Semantic Reasoning - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset📊LeaderBoard
• CETBench (2025): CETBench: A Novel Dataset constructed via Transformations over - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
• ICPC-Eval (2025): ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• CoQuIR (2025): CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
• OJBench (2025): OJBench: A Competition Level Code Benchmark For Large Language Models - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
• CORE (2025): CORE: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Tasks - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
• CLMEEval (2025): Model Editing for LLMs4Code: How Far are We? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• LONGCODEU (2025): LONGCODEU: Benchmarking Long-Context Language Models on Long Code Understanding - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge
• LiveRepoReflection (2025): Turning the Tide: Repository-based Code Reflection - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🌐Website 📊LeaderBoard
• LoCoBench (2025): LoCoBench: A Benchmark for Long-Context Large Language Models in Complex Software Engineering - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github
• CodeFuse-CR-Bench (2025): CodeFuse-CR-Bench: A Comprehensiveness-aware Benchmark for End-to-End Code Review Evaluation in Python Projects - task: End-to-end code review - granularity: Repository / Workflow - interaction: Agentic - evaluation: LLM-as-Judge / Human
• ContextCRBench (2025): Benchmarking LLMs for Fine-Grained Code Review with Enriched Context in Practice - task: Fine-grained code review - granularity: File / Repository - interaction: Single-turn / Agentic - evaluation: LLM-as-Judge / Human
• CodeJudgeBench (2025): CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks - task: Code judging / evaluation - granularity: Function / Repository - interaction: Single-turn - evaluation: LLM-as-Judge / Human - 🤗Dataset
• CRUXEval (2024): CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 📊LeaderBoard
• Poor-CodeSumEval (2024): How Effectively Do Code Language Models Understand Poor-Readability Code? - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• CodeScope (2024): CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 📊LeaderBoard
  🤗Dataset
• GenCodeSearchNet (2023): GenCodeSearchNet: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding - task: Code understanding / reasoning / search - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset

• CodegenBench (2026): CodegenBench: Can LLMs Write Efficient Code Across Architectures? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Code Dataset
• ENAMEL (2025): How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
• Improving Assembly Code Performance with Large Language Models via Reinforcement Learning (2025): Improving Assembly Code Performance with Large Language Models via Reinforcement Learning - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
• EFFIBENCH-X (2025): EFFIBENCH-X:A Multi-Language Benchmark fo rMeasuring Effciency ofLLM.Generated Code - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
• PERFFORGE (2025): Synthesizing Performance Constraints for Evaluating and Improving Code Efficiency - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
• SWE-Perf (2025): SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset 🌐Website
• TRACY (2025): TRACY: Benchmarking Execution Efficiency of LLM-Based Code Translation - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance
• EvalPerf (2024): Evaluating Language Models for Efficient Code Generation - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset 🌐Website
• EffiBench (2024): EffiBench: Benchmarking the Efficiency of Automatically Generated Code - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github
• Mercury (2024): Mercury: A Code Efficiency Benchmark for Code Large Language Models - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
• ECCO (2024): ECCO: Can We Improve Model-Generated Code Efficiency Without Sacrificing Functional Correctness? - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🤗Dataset
• PIE (2024): Learning Performance-Improving Code Edits - task: Performance optimization - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution / Performance - Github 🌐Website

• BigDocs-Bench (2025): BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - 🤗Dataset
  🌐Website
• WebCode2M (2025): WebCode2M: A Real-World Dataset for Code Generation from Webpage Designs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
  🤗Dataset
• Design2Code (2025): Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• DiagramGenBenchmark (2025): From Words to Structured Visuals: A Benchmark and Framework for Text-to-Diagram Generation and Editing - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
  🤗Dataset
• ChartMimic (2025): ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website 🤗Dataset
• SVG-Bench (2025): StarVector: Generating Scalable Vector Graphics Code from Images and Text - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
  🤗Dataset
• LLM4SVG (2025): Empowering LLMs to Understand and Generate Complex Vector Graphics - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
• ChartCoder (2025): ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Code-Vision (2025): Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
• Flame-React-Eval (2025): Advancing vision-language models in front-end development via data synthesis - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• vTikZ (2025): LLM Code Customization with Visual Results: A Benchmark on TikZ - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
• Plot2XML (2025): Draw with Thought: Unleashing Multimodal Reasoning for Scientific Diagram Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
• Flow2Code (2025): Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github
• DesignBench (2025): DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• WebUIBench (2025): WebUIBench: A Comprehensive Benchmark for Evaluating Multimodal - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• FrontendBench (2025): FrontendBench: A Benchmark for Evaluating LLMs on Front-End Development via Automatic Evaluation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge
• ArtifactsBench (2025): ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website 🤗Dataset📊 Leaderboard
• M^2 EVAL (2025): Multilingual Multimodal Software Developer for Code Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• WebGen-Bench (2025): WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• MMCode (2024): MMCode: Benchmarking Multimodal Large Language Models for Code Generation with Visually Rich Programming Problems - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Drawing Pandas (2024): Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Web2Code (2024): Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
  🌐Website
• VGBench (2024): VGBench: Evaluating Large Language Models on Vector Graphics Understanding and Generation - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• SVGEditBench (2024): SVGEditBench: A Benchmark Dataset for Quantitative Assessment of LLM's SVG Editing Capabilities - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Plot2Code (2024): Plot2Code: A Comprehensive Benchmark for Evaluating Multi-modal Large Language Models in Code Generation from Scientific Plots - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• HumanEval-V (2024): HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
  📊LeaderBoard
  🤗Dataset
• WebSight-Test (2024): WAFFLE: Multi-Modal Model for Automated Front-End Development - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Sketch2Code (2024): Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
• Interaction2Code (2024): Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
  📊LeaderBoard
• ScratchEval (2024): ScratchEval: Are GPT-4o Smarter than My Child? Evaluating Large Multimodal Models with Visual Programming Challenges - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• MRWeb (2024): MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• Image2Struct (2024): Image2Struct: Benchmarking Structure Extraction for Vision-Language Models - task: Frontend / visual-interactive generation - granularity: Project / UI - interaction: Multi-turn / Agentic - evaluation: Execution / LLM-as-Judge - Github 🌐Website
  🤗Dataset

• DyCodeEval (2025): DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• BigCodeBench (2025): BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
• EvoCodeBench (2025): EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• LibEvolutionEval (2025): LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation - task: Version-specific code completion - granularity: Function / Repository - interaction: Single-turn - evaluation: Execution - Github 🌐Website
• DynaCode (2025): DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs (2025): A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• LeetCodeDataset (2025): LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• CodeFlowBench (2025): CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• CodeMixBench (2025): CodeMixBench: Evaluating Large Language Models on Code Generation with Code-Mixed Prompts - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - 🤗Dataset
• CPRet (2025): CPRet: A Dataset, Benchmark, and Model for Retrieval in Competitive Programming - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
• ELABORATION (2025): ELABORATION: A Comprehensive Benchmark on Human-LLM Competitive Programming - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
• OSS-Bench (2025): OSS-Bench: Benchmark Generator for Coding LLMs - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
• VERINA (2025): VERINA: Benchmarking Verifiable Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• OIBench (2025): OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - 🤗Dataset
• IFEvalCode (2025): IFEvalCode: Controlled Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🌐Website
• CodeEval Pro (2025): HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
• Code2Bench (2025): Dynamic Benchmark Construction for Evaluating Large Language Models on Real-World Codes - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🌐Website
• STEPWISE-CODEX-Bench (2025): STEPWISE-CODEX-Bench: Evaluating Complex Multi-Function Comprehension and Fine-Grained Execution Reasoning - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• AutoCodeBench (2025): AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 🌐Website 📊LeaderBoard
• SR-Eval (2025): SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• DataSciBench (2025): DataSciBench: An LLM Agent Benchmark for Data Science - task: Data science code generation - granularity: Project / Workflow - interaction: Single-turn / Agentic - evaluation: Execution - Github
• Spider 2.0 (2025): Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows - task: Text-to-SQL - granularity: Query / Database - interaction: Single-turn - evaluation: Execution - Github 🌐Website
• CRUST-Bench (2025): CRUST-Bench: A Comprehensive Benchmark for C-to-safe-Rust Transpilation - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github Dataset
• Paper2Code (2025): Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning - task: Research code generation - granularity: Project / Repository - interaction: Agentic - evaluation: Execution / LLM-as-Judge - Github 🤗Dataset
• PPM (2024): PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• RepoBench (2024): RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• CatCoder (2024): Enhancing Repository-Level Code Generation with Integrated Contextual Information - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• StudentEval (2024): StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• DevEval (2024): DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• CoderEval (2024): CoderEval: A Benchmark of Pragmatic Code Generation with Generative Pre-trained Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github
• ConCodeEval (2024): ConCodeEval: Evaluating Large Language Models for Code Constraints in Domain-Specific Languages - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• CodeScope (2024): CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 📊LeaderBoard
  🤗Dataset
• OOP (2024): OOP: Object-Oriented Programming Evaluation Benchmark for Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• L2CEval (2024): L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• HumanExtension (2024): Exploring Language Model's Code Generation Ability with Auxiliary Functions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• LLM4Decompile (2024): LLM4Decompile: Decompiling Binary Code with Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• PYCOMMITS (2024): Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• CodeAgentBench (2024): CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution
• SAFIM (2024): Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• RTLLM (2024): RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
• OpenLLM-RTL (2024): OpenLLM-RTL: Open Dataset and Benchmark for LLM-Aided Design RTL Generation - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
• MultiPL-E (2023): MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• MCoNaLa (2023): MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• LCC (2023): LongCoder: A Long-Range Pre-trained Language Model for Code Completion - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• CodeClarQA (2023): Python Code Generation by Asking Clarification Questions - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• EvalPlus (2023): Is Your Code Generated by Chat{GPT} Really Correct? Rigorous Evaluation of Large Language Models for Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset 📊LeaderBoard
• CrossCodeEval (2023): CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• ODEX (2023): Execution-Based Evaluation for Open-Domain Code Generation - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• DS-1000 (2023): DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation - task: Data science code generation - granularity: Project / Workflow - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset 🌐HomePage
• BIRD (2023): Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs - task: Text-to-SQL - granularity: Query / Database - interaction: Single-turn - evaluation: Execution - Github 🌐Website
• CodeTransOcean (2023): CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
• VerilogEval (2023): VerilogEval Evaluating Large Language Models for Verilog Code Generation - task: Domain-specific code generation - granularity: Project / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github 🤗Dataset
• CodeContests (2022): Competition-Level Code Generation with AlphaCode - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github Dataset
• HumanEval (2021): Evaluating Large Language Models Trained on Code - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• MBPP (2021): Program Synthesis with Large Language Models - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• APPS (2021): Measuring Coding Challenge Competence With APPS - task: Code generation / completion - granularity: Function / Repository - interaction: Single-turn / Multi-turn - evaluation: Unit Tests / Execution - Github 🤗Dataset
• TransCoder (2020): Unsupervised Translation of Programming Languages - task: Code translation / migration - granularity: Function / Repository - interaction: Single-turn / Agentic - evaluation: Execution - Github(deprecated) Github(new) Dataset