Clinical-R1-3B: A reinforcement learning-based model inspired by DeepSeek-R1, designed to enhance medical question-answering and reasoning capabilities.
This repository is based on veRL and Qwen-2.5-3B.
All datasets and models are available on the HF Collection.
conda env create -f environment.yaml -n verl
conda activate verl
pip install -e .
./scripts/data_preparation/raw2csv.sh
./scripts/data_preparation/grpo.sh
./scripts/data_preparation/simple_sft.sh
For distillation (cold start) data preparation, you need to fill in your Deepseek API key and conda environment path into ./scripts/data_preparation/deepseek_api_call.sh, and run
./scripts/data_preparation/deepseek_api_call.sh
After all screens are done with API calls, run
./scripts/data_preparation/distil_sft.sh
./scripts/sft/run_qwen_3b_peft_simple_sft.sh
./scripts/sft/run_qwen_3b_peft_deepseek_distil.sh
Before running GRPO-related training, we need to process the merge the peft models:
./scripts/merge_lora.sh
./scripts/grpo/run_qwen-3B_seq_balance_medqa.sh
./scripts/grpo/run_qwen-3B_after-deepseek-distil-on-medqa_seq_balance_medqa.sh
./scripts/evaluation/model_response_generation.sh
./scripts/evaluation/model_response_evaluation.sh
./scripts/evaluation/cognitive_behavior_evaluation.sh