Code-A1 jointly trains a Code LLM and a Test LLM with opposing objectives, enables white-box adversarial test generation without self-collusion, and uses a Mistake Book replay mechanism to retrieve historical failure cases.

• Motivation
• Highlights
• Repository Structure
• Installation
• Dataset
• Quick Start
• Main Results
• Citation
• Acknowledgement

Reinforcement learning for code generation typically depends on unit-test pass rates as verifiable rewards. In practice, this creates three persistent issues:

• Static golden tests are limited in coverage and quickly saturate as the code model improves.
• Black-box generated tests are often too generic to expose implementation-specific bugs.
• Single-model self-play introduces self-collusion: the same model can generate easy tests that inflate rewards without improving code quality.

Code-A1 addresses this by separating the two roles. A Code LLM is optimized to solve programming problems, while a Test LLM is optimized to expose errors in the generated code. This makes white-box test generation useful rather than dangerous: the Test LLM can inspect candidate implementations and generate targeted adversarial tests. The framework further stabilizes co-evolution with a Mistake Book replay mechanism and a composite reward that balances test validity and adversarial difficulty.

• Adversarial co-evolution: jointly optimizes a Code LLM and a Test LLM with opposite objectives instead of collapsing both roles into one model.
• White-box test generation without self-collusion: the Test LLM conditions on candidate code to synthesize implementation-specific attack tests.
• Mistake Book replay: historical failure cases are replayed during training so the Code LLM does not forget earlier weaknesses.
• Composite reward for Test LLM: balances executable test validity with adversarial difficulty.
• Strong empirical gains: improves both code generation and test generation, and generated tests can even serve as competitive static supervision.

The project uses Python 3.10 and uv for environment management. The provided scripts create two separate environments: one for RL training and one for evaluation.

cd Code-A1
bash rl_env.sh
source .venv-rl/bin/activate

This script creates .venv-rl, installs the local verl fork in editable mode, and installs the main runtime dependencies including ray, vllm/sglang support, and sandbox_fusion.

cd Code-A1
bash eval_env.sh
source .venv-eval/bin/activate

This script creates .venv-eval and installs the packages needed for BigCodeBench evaluation, mutation-based tooling, vllm==0.11.0, wandb, and sandbox_fusion.

Before training or evaluation, configure environment variables in Code-A1/code/rl/run/set_env.sh:

export SANDBOX_FUSION_ENDPOINT=YOUR_SANDBOX_IP
export WANDB_API_KEY=YOUR_WANDB_API_KEY

The training scripts also expect:

• A reachable sandbox_fusion service for secure code execution.
• GPUs compatible with the selected model scale and FSDP-based training.
• Access to the base models specified in the YAML configs, such as Qwen/Qwen2.5-Coder-1.5B-Instruct.

The training data is stored in Code-A1/code/rl/train_data/kodcode_hard_dual_model_training_data_mix.parquet and is built from 9,688 hard-difficulty questions from KodCode-V1.

Validate that the sandbox executor is reachable before running training:

cd Code-A1
bash test_sandbox.sh

Expected output should contain a successful RunCodeResponse with return_code=0.

Provided launch scripts:

cd Code-A1
source .venv-rl/bin/activate

bash code/rl/run/1.5B_A1.sh
bash code/rl/run/3B_A1.sh
bash code/rl/run/7B_A1.sh

For example, the default 1.5B config uses:

• Qwen/Qwen2.5-Coder-1.5B-Instruct as the Code LLM
• Qwen/Qwen2.5-Coder-1.5B-Instruct as the Test LLM
• alpha: 0.5 in the composite test reward
• n_gpus_per_node: 4 for both Code LLM and Test LLM
• rollout.n: 8 during training and n: 32 during validation sampling

For the Code LLM:

• HumanEval+ and MBPP+ are included in the validation data and are evaluated during training.
• BigCodeBench is evaluated separately with:

cd Code-A1
source .venv-eval/bin/activate
bash code/rl/run/eval.sh

For the Test LLM, evaluation is conducted with UnLeakedTestBench

Code-A1 outperforms both the Golden Tests baseline trained on human annotations and the Self-Play approach on HumanEval+, MBPP+, and BigCodeBench.

The Test LLM also improves substantially under adversarial co-evolution. Notably, the 3B Test LLM trained with Code-A1 reaches Mul = 15.29, exceeding the unoptimized 7B base model (14.72).

@misc{wang2026codea1adversarialevolvingcode,
      title={Code-A1: Adversarial Evolving of Code LLM and Test LLM via Reinforcement Learning}, 
      author={Aozhe Wang and Yuchen Yan and Nan Zhou and Zhengxi Lu and Weiming Lu and Jun Xiao and Yueting Zhuang and Yongliang Shen},
      year={2026},
      eprint={2603.15611},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.15611}, 
}

The RL training stack is built on top of the excellent verl framework, which is included in this repository under Code-A1/verl. Many thanks to the verl team for open-sourcing the infrastructure that this project extends.