RRC: Open-ended Reinforcement Learning via Ranking-based Reward Construction

This repository contains the code for RRC, an open-ended reinforcement learning approach. RRC focuses on reward construction in open-ended scenarios: instead of relying on fixed ground-truth answers to evaluate generated responses, it leverages an external generative reward model to perform relative comparisons among candidate responses and transforms the resulting preference signals into rewards for GRPO and other RL algorithms. The paper is available at this link.

RRC works inside the reward construction stage. For each prompt, we sample multiple completions from the current policy model. The external RRC plugin receives these completions and computes rewards as follows:

1. Extract the final assistant response from each sampled completion.
2. Use the dataset solution field as the original user instruction.
3. Group completions that share the same instruction.
4. Ask an external reward model to compare Response A and Response B.
5. Aggregate the pairwise preference signal through ranking, majority voting, reference-baseline wins, or confidence scores.
6. Return one scalar reward per completion to the SWIFT GRPO trainer.

The reward model service is expected to follow the OpenAI-compatible Chat Completions API. The default generative RM prompt asks the reward model to choose A or B and output the decision in an answer tag.

• Installation
• Preparing Models and Datasets
• Reward Plugins
• Training Scripts
• Using RRC in RLHF
• Troubleshooting
• Citation
• Acknowledgement

This repository is modified from MS-SWIFT. If you encounter installation issues related to PyTorch, CUDA, vLLM, or distributed training, first check the upstream SWIFT documentation and issues.

git clone <your-rrc-repo-url>
cd RRC

pip install -e .
pip install aiohttp vllm

For optional SWIFT features:

pip install -e ".[all]"

Use any SWIFT-supported causal language model as the policy model. For example:

Qwen/Qwen2.5-7B-Instruct

RRC requires an external reward model that can judge which response is better for a given instruction. The service must expose an OpenAI-compatible API:

• GET /v1/models
• POST /v1/chat/completions

For probability-based or entropy-based reward plugins, the service must also return logprobs.

Before training, configure the reward-model endpoint with environment variables:

export RRC_RM_HOST=127.0.0.1
export RRC_RM_PORTS=8200,8201,8202,8203
export RRC_RM_MODEL=rrc-rm
export RRC_RM_TIMEOUT=120

For reference-baseline plugins, configure both the reward model and the reference model:

export RRC_REF_HOST=127.0.0.1
export RRC_REF_PORTS=8100,8101,8102,8103
export RRC_REF_MODEL=rrc-ref
export RRC_REF_N=8

Each training item should contain the conversation input and a solution field. In RRC, solution stores the original user instruction so that completions can be grouped and compared.

Minimal JSONL example:

{"messages": [{"role": "user", "content": "Write a concise answer about why exploration matters in RL."}], "solution": "Write a concise answer about why exploration matters in RL."}

Recommended response format:

<think>
reasoning process
</think>
<answer>
final answer
</answer>

The format reward checks whether the completion contains both tag pairs.

The RRC reward plugins are placed in the SWIFT source tree:

swift/rrc_plugins/rrc.py

Load this file with --external_plugins swift/rrc_plugins/rrc.py. The plugin provides compact reward names for ranking, voting, reference-baseline comparison, preference-token probability, confidence-aware scoring, and response-format checking.

When using SWIFT server mode, start the rollout service before training:

CUDA_VISIBLE_DEVICES=0,1 swift rollout \
  --model Qwen/Qwen2.5-7B-Instruct \
  --vllm_tensor_parallel_size 2 \
  --vllm_data_parallel_size 1 \
  --port 8000

Example command with vLLM:

CUDA_VISIBLE_DEVICES=2 vllm serve <your-generative-rm-path> \
  --served-model-name rrc-rm \
  --host 0.0.0.0 \
  --port 8200

If you use multiple RM instances, start one service per port and list all ports in the plugin config.

CUDA_VISIBLE_DEVICES=0,1,2,3 \
NPROC_PER_NODE=4 \
swift rlhf \
  --rlhf_type grpo \
  --model Qwen/Qwen2.5-7B-Instruct \
  --dataset /path/to/train.jsonl \
  --external_plugins swift/rrc_plugins/rrc.py \
  --reward_funcs rrc_vote rrc_format \
  --reward_weights 1.0 0.1 \
  --use_vllm true \
  --vllm_mode server \
  --vllm_server_host 127.0.0.1 \
  --vllm_server_port 8000 \
  --torch_dtype bfloat16 \
  --tuner_type full \
  --max_completion_length 2048 \
  --num_generations 8 \
  --temperature 1.0 \
  --top_p 0.9 \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 2 \
  --learning_rate 1e-6 \
  --num_train_epochs 1 \
  --logging_steps 1 \
  --save_steps 500 \
  --save_total_limit 2 \
  --log_completions true \
  --deepspeed zero2 \
  --beta 0.04

For colocate mode, replace the server-mode rollout arguments with:

--use_vllm true \
--vllm_mode colocate \
--vllm_gpu_memory_utilization 0.6 \
--sleep_level 1 \
--offload_model true \
--offload_optimizer true

The basic form of RRC computes rewards from pairwise response comparisons:

def compute_pair_rewards(instruction, response_a, response_b):
    # Ask the reward model which response is better.
    judgment = reward_model_compare(instruction, response_a, response_b)

    if judgment == "A":
        return 1.0, 0.0
    if judgment == "B":
        return 0.0, 1.0
    return 0.0, 0.0

For each prompt, RRC can rank multiple sampled responses using repeated pairwise comparisons:

def list_wise_response_ranking(instruction, responses):
    # Rank responses from best to worst with pairwise RM comparisons.
    ranked_responses = pairwise_sort(instruction, responses)
    rewards = rank_to_reward(ranked_responses)
    return rewards

This is the default idea behind the ranking and voting reward variants.

The reference-baseline variant first generates responses from a fixed reference model. A policy response receives a higher reward if it beats more reference responses:

def reference_baseline_reward(instruction, policy_response, reference_responses):
    wins = 0
    for reference_response in reference_responses:
        reward_policy, reward_ref = compute_pair_rewards(
            instruction,
            policy_response,
            reference_response,
        )
        if reward_policy > reward_ref:
            wins += 1
    return float(wins)

This is useful for open-ended tasks where there is no single correct answer, but a fixed reference policy can define a moving comparison target.

Before launching a full training job, test the RM connection with a small script:

from swift.rrc_plugins.rrc import RRCVoteReward

orm = RRCVoteReward()
rewards = orm(
    completions=[
        "<think>...</think><answer>Response 1</answer>",
        "<think>...</think><answer>Response 2</answer>",
    ],
    solution=[
        "Write a concise answer about why exploration matters in RL.",
        "Write a concise answer about why exploration matters in RL.",
    ],
)
print(rewards)

• --external_plugins: imports swift/rrc_plugins/rrc.py and registers the RRC reward functions.
• --reward_funcs: selects reward functions by registry name.
• --reward_weights: sets reward weights in the same order as --reward_funcs.
• --num_generations: controls how many completions are sampled per prompt. Values greater than 2 are recommended.
• --log_completions true: saves prompts, completions, and rewards for debugging.
• --top_entropy_quantile and --log_entropy: enables SWIFT's entropy-mask and entropy-logging features.

Check the following:

• The plugin host and ports point to running reward-model services.
• The --reward_funcs name exactly matches the plugin's orms[...] registry name.
• The reward model response contains <answer>A</answer> or <answer>B</answer>.
• The dataset solution field contains the original user instruction.
• The completion contains <answer>...</answer> if you rely on answer extraction.

RRC adds extra reward-model inference calls. Pairwise comparison, voting, and reference-baseline variants are more expensive than standard accuracy rewards. To speed up early experiments, reduce:

• num_generations
• vote_n in the plugin
• ref_n in the plugin
• RM max_tokens

You can also run more reward-model service ports and list them in VLLMConfig.ports.

If this repository is useful for your research, please cite this project and the upstream MS-SWIFT framework.

@misc{wang2026rrcunlockinggenerativereward,
      title={RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction}, 
      author={Chenglong Wang and Ziming Zhu and Yifu Huo and Bei Li and Qiaozhi He and Yan Ding and Xiaoyang Hao and Yuxin Gao and Tianhua Zhou and Xiaojia Chang and Tongran Liu and Jingbo Zhu},
      year={2026},
      eprint={2608.06310},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2608.06310}, 
}

This project is built on the excellent ModelScope SWIFT codebase. We also thank the open-source reward-model and RLHF communities for making reproducible research in preference learning possible.