This document presents the results of CPPO on the verl framework. Verl is a flexible, efficient, and production-ready RL training library for large language models (LLMs). Compared to the OpenR1 framework, verl supports larger batch sizes under the same GPU memory constraints. Therefore, we use larger batch sizes in verl than in OpenR1. Due to differences in frameworks and training parameters, the accuracy and acceleration ratios of CPPO on verl may differ from those on OpenR1. With a suitable pruning rate, CPPO achieves significant acceleration without sacrificing accuracy compared to GRPO on verl. However, an excessively high pruning rate may remove high-quality completions and reduce training effectiveness, which is consistent with observations on the OpenR1 framework.

Benefiting from the joint optimization of the verl framework and the CPPO algorithm, the training time for CPPO has been reduced to 1932.56s (k=2) without sacrificing accuracy compared to GRPO. In contrast, under the OpenR1 framework, the training time for CPPO is 2813s even with k=1.

Benefiting from the joint optimization of the verl framework and the CPPO algorithm, the training time for CPPO has been reduced to 12652s (k=8) without sacrificing accuracy compared to GRPO. In contrast, under the OpenR1 framework, the training time for CPPO is 12959s even with k=4.

cd cppo_verl/
conda create -n verl python=3.10
conda activate verl
pip3 install torch==2.6.0
pip3 install flash-attn --no-build-isolation
pip3 install -e .
pip3 install vllm==0.8.2
pip3 install math_verify

You need two GPU with 80G memory to reproduce our results on GSM8K.

bash recipe/cppo/gsm8k_grpo.sh

bash recipe/cppo/gsm8k_cppo.sh

Before evaluation, convert the model to Hugging Face format using the conversion script.

bash recipe/cppo/gsm8k_eval.sh

You need four GPU with 80G memory to reproduce our results on Math.

bash recipe/cppo/math_grpo.sh

bash recipe/cppo/math_cppo.sh

Before evaluation, convert the model to Hugging Face format using the conversion script.

bash recipe/cppo/math_eval.sh