SOAR

This is the official implementation for the paper "SOAR: Supervision from Observation for Agentic Reinforcement Learning" (ACL 2026).

SOAR improves agentic RL by giving entropy-weighted positive advantages to observation tokens, allowing agents to learn environment dynamics from confident actions instead of relying only on sparse rewards.

The training pipeline consists of two stages:

1. Stage 1 — Supervised Fine-Tuning (SFT) (Optional): Uses LLaMA-Factory for full-parameter fine-tuning with DeepSpeed ZeRO-3.
2. Stage 2 — Reinforcement Learning with GRPO: Uses a modified verl framework with agent-tool interaction support and the Observation Loss mechanism.

SOAR/
├── tools/                    # Tool server infrastructure
│   ├── scripts/              # Data download and preprocessing
│   ├── wiki_server/          # Local dense retrieval server (FAISS + e5)
│   └── web_server/           # Google/Serper search proxy server
├── sft/                      # Stage 1: Supervised Fine-Tuning
│   ├── sft_scripts/          # Training scripts and YAML configs
│   ├── src/                  # LLaMA-Factory framework
│   └── data/                 # SFT training data
├── rl/                       # Stage 2: Reinforcement Learning
│   ├── scripts/              # Training launch scripts and configs
│   ├── verl/                 # Modified verl framework with agent support
│   └── merge_ckpt/           # Checkpoint conversion utilities
└── evaluation/               # Evaluation pipeline
    ├── src/                  # Evaluation source code
    └── vllm_scripts/         # Model serving scripts

The local retriever uses the intfloat/e5-base-v2 encoder with a FAISS index for dense retrieval over the wiki-18 corpus. The server runs on port 8000 with configurable GPU allocation.

First, create a separate environment for the local retriever:

conda create -n local-retriever python=3.10
conda activate local-retriever

conda install pytorch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 pytorch-cuda=12.1 -c pytorch -c nvidia
pip install -r tools/wiki_server/requirements.txt

conda install -c pytorch -c nvidia faiss-gpu=1.8.0

Second, download the retriever corpus and index:

cd SOAR/tools
save_path=/the/path/to/save
python scripts/download.py --save_path $save_path
cat $save_path/part_* > $save_path/e5_Flat.index
gzip -d $save_path/wiki-18.jsonl.gz
tar -xvf wiki-18.jsonl

Finally, launch the retriever:

bash launch_rag_server.sh -i <num_gpus>

The web search server is a FastAPI proxy for the Serper Google Search API. It includes persistent pickle-based caching to minimize API calls and runs on port 9001.

First, setup the serper search api key in environment_serper.sh. Then, set environment variables via

cd tools/web_server
source environment_serper.sh

Finally, start the webserver:

bash start_serper_server.sh start

If you want to stop the webserver, run:

bash start_serper_server.sh stop

The summarization model (Qwen3-30B-A3B) is used to summarize search results during both training and evaluation. It extracts key information from raw search results and produces a concise "Final Information" summary that the agent can reason over.

Create a vllm environment for model serving:

conda create -n vllm python=3.10

pip install vllm==0.10.1

launch local Qwen3-30B-A3B model via:

cd rl
conda activate vllm
bash scripts/vllm_launch_summarize_model_cuda0-1.sh

conda create -n soar-sft python=3.10
cd sft

pip install -r requirements.txt
pip install -e .

Download your SFT dataset from ToolStar-SFT and place it in sft/data/toolstar_sft.json. Launch training via:

cd sft_scripts/
bash sft_train_3b_4c.sh

The RL stage uses GRPO (Group Relative Policy Optimization) with the Observation Loss mechanism.

cd rl

conda create soar-rl python=3.10
conda activate soar-rl

pip3 install torch==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip3 install flash-attn==2.7.4.post1 --no-build-isolation

pip install -r requirements.txt

cd verl
pip install -e .

These experiments use the local wiki retriever + Python tool. The training scripts are at scripts/3B_wiki_soar.sh and scripts/7B_wiki_soar.sh.

In the script, setup your CONDA_PATH, ACTOR_MODEL_PATH (base model path), SERVER_HOST (local search server ip, usually localhost), and SUMM_HOST (summary model url address, usually localhost).

Then, launch training via:

bash scripts/3B_wiki_soar.sh
# or
bash scripts/7B_wiki_soar.sh

These experiments use Google web search + Python tool .

The training scripts are at scripts/8B_wiki_soar.sh and scripts/14B_wiki_soar.sh.

In the script, setup your CONDA_PATH, ACTOR_MODEL_PATH (base model path), SEARCH_SERVER_URL (web search server ip), and SUMM_HOST (summary model url address, usually localhost).

Then, launch training via:

bash scripts/8B_wiki_soar.sh
# or
bash scripts/14B_wiki_soar.sh

After RL training, your model checkpoint is stored in fsdp format. If you want to convert it to hf format, set the checkpoint path and target path in rl/merge_ckpt/convert_checkpoint_from_verl_to_hf.sh and run it.

Create a eval environment for evaluation:

conda create -n soar-eval python=3.10
cd evaluation
pip install -r requirements.txt

# launch summarization model (you can also use the summarization model launched for training)
bash vllm_launch_summarize_model_cuda0-1.sh

# launch model to be evaluated
bash vllm_launch_reasoning_model_cuda2-3.sh

In infer_agent_traj.sh, set the inference_endpoints, MODEL_PATH, API_KEYS, DEFAULT_MODEL, SUMM_MODEL_URLS based on your launched model. Also, setup the CONDA_PATH for python tool call. Launch inference via:

bash infer_agent_traj.sh

First, setup the LLM-as-Judge model's url path in the API_BASE_URL variable in evaluate.sh. Metrics are calculated by running:

bash evaluate_output.sh <inference-output-path>

@inproceedings{li2026soar,
  title={SOAR: Supervision from Observation for Agentic Reinforcement Learning},
  author={Meng Li, Lei Li, Xiting Wang, Yi Yuan, Zheng Wei, Bruce Bian, Zang Li},
  booktitle={Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year={2026}
}