This is a fork of SkyRL maintained for the Marin project (marin-community), where it powers agentic RL training (SkyRL + Harbor). It was originally developed for the OpenThoughts-Agent project; that line of work now continues here under Marin.

MarinSkyRL is maintained as a hard snapshot; no upstream sync or merge-back is planned.

The repository root builds one marinskyrl wheel containing the typed Iris launcher, skyrl_train, and skyrl_gym. Its base dependency set is CPU-only and supports launcher inspection, validation, and dry runs:

uv sync --frozen
uv run --frozen marinskyrl --help

The native Tinker reasoning reproduction records the step-400 SFT adapter, the one-step OPD gate, AIME 2024 evidence, and replay commands.

Python extras cannot replace a base CPU Torch wheel with a CUDA wheel, so cpu and cuda are mutually exclusive wheel profiles. GPU-only component extras imply cuda: ordinary training commands select only vllm, while the standard image adds fsdp for its TorchTitan expert-parallel path and optimized attention kernels used by hybrid architectures such as Qwen3.5. The Megatron image adds megatron. The GPU profiles use Torch 2.13.0+cu132, and vllm selects immutable Linux x86_64 H100 and aarch64 GB200 wheels. The optional fsdp native extensions and all megatron native wheels remain x86_64-only; the aarch64 Grug policy path uses eager attention. Iris installs the frozen root profile before launch. The cpu profile remains on Torch 2.11.0.

The walkthrough below reproduces the original OpenThoughts-Agent v1 release (kept here for reference), i.e.:

• Using open-thoughts/OpenThinker-Agent-v1-SFT as base
• GRPO with the data open-thoughts/OpenThoughts-Agent-v1-RL, while
• Evaluating with open-thoughts/OpenThoughts-TB-dev, and
• Getting the final open-thoughts/OpenThinker-Agent-v1

Install SkyRL

conda create -n otagent python=3.12
conda activate otagent
pip install --index-url https://download.pytorch.org/whl/cu128 torch==2.7.1 torchvision
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.0.post2/flash_attn-2.8.0.post2+cu12torch2.7cxx11abiFALSE-cp312-cp312-linux_x86_64.whl

git clone https://github.com/mlfoundations/SkyRL
cd SkyRL/skyrl-train/
pip install -e .
pip install "vllm==0.10.1.1"
cd ../..

Install Harbor

git clone https://github.com/CharlieFRuan/harbor
cd harbor
git checkout 112425-terminus2-messages
pip install -e .

Remainings

pip install fastapi uvicorn

We will soon make things uv-syncable.

conda activate otagent
# Download the eval dataset (OTTB-dev)
hf download open-thoughts/OpenThoughts-TB-dev --repo-type=dataset
# Download the train dataset
hf download open-thoughts/OpenThoughts-Agent-v1-RL --repo-type=dataset
# cd into the downloaded folder, say /path/to/.cache/huggingface/hub/datasets--open-thoughts--OpenThoughts-Agent-v1-RL/snapshots/hash_code
cd /path/to/.cache/huggingface/hub/datasets--open-thoughts--OpenThoughts-Agent-v1-RL/snapshots/hash_code
python extract_parquet_tasks.py tasks_new.parquet ./extracted_tasks

Then configure the paths and API keys at the top of the script, and run:

cd SkyRL/skyrl-train
bash run_marinskyrl.sh

The script is designed to run on 8 GPUs single-node. If that is not your setup, modify these configs correspondingly:

  trainer.placement.policy_num_nodes=1 \
  trainer.placement.ref_num_nodes=1 \
  trainer.placement.policy_num_gpus_per_node=8 \
  trainer.placement.ref_num_gpus_per_node=8 \
  generator.num_inference_engines=8 \
  generator.inference_engine_tensor_parallel_size=1 \