RPent (Recursive Physical Agent) is an open framework for building embodied agents that continuously evolve through recursive interaction with the physical world. Rather than prescribing a single foundation model, RPent provides a recursive agent framework that harnesses heterogeneous intelligence, including perception, reasoning, memory, execution, and self-evolution, into a unified physical agent. Through continuous interaction, reflection, and adaptation, RPent enables physical agents to acquire new capabilities and evolve beyond their initial design.
RPent is built upon three core design principles: service-oriented, standardized, and composable. RPent enables capabilities to be deployed as reusable services, connected through unified interfaces, and flexibly composed into diverse physical agents. Together, these principles allow RPent to move beyond traditional robot control frameworks and establish an agentic infrastructure for the physical world, where intelligence is not only deployed, but continuously built, expanded, and evolved.
- [2026/07] 🔥 Our first RPent publication, Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents, is released.
| Agentic Planner | Action Primitive | Simulator | Real World |
|---|---|---|---|
|
|
|
|
1. Install RPent with a single pip install.
git clone https://github.com/RLinf/RPent rpent && cd rpent
pip install -e ".[full]".[full] is the default end-to-end stack (openpi Pi0.5 VLA + LIBERO-PRO simulator + SAM 3.0 on the RLinf runtime).
If you don't need the whole stack, see the installation docs for narrower extras.
2. Download the LIBERO-PRO simulator assets.
liberopro-download-assets --skip-existing💡 Slow connection to Hugging Face? Download through the mirror:
HF_ENDPOINT=https://hf-mirror.com liberopro-download-assets --skip-existing.
See the installation docs for other simulators.
3. Configure keys and checkpoints, then run.
# Anthropic key; no need to export the base url if you use the official endpoint.
export ANTHROPIC_BASE_URL=https://xxx
export ANTHROPIC_API_KEY=sk-xxx
# VLA checkpoint — download from
# https://huggingface.co/RLinf/RLinf-Pi05-LIBERO-130-fullshot-SFT
export PI05_CHECKPOINT_PATH=/path/to/rlinf-pi05-libero-130-fullshot-sft
# SAM 3.0 checkpoint — download from
# https://huggingface.co/facebook/sam3
# https://modelscope.cn/models/facebook/sam3
export SAM3_CHECKPOINT_PATH=/path/to/sam3/sam3.pt
export LIBERO_TYPE=pro
export CUDA_VISIBLE_DEVICES=0
# Run one task: libero_object_swap, task 2, seed 0, using the `claude_code` planner.
rpent --env libero --suite libero_object_swap --task 2 --seed 0 \
--planner claude_code --model claude-opus-4-8See the planner docs to configure other planners (api, codex) and model providers.
Add --interactive (-i) to steer the agent live from your terminal. At the you> prompt, the built-in task is pre-filled — press Enter to use it or replace it with your own — then type any message while it runs to steer the agent at the next turn (/help lists commands; /quit or Ctrl-D ends). Requires an interactive terminal (TTY).
rpent --env libero --suite libero_object_swap --task 2 --seed 0 \
--planner claude_code --model claude-opus-4-8 --interactiveAdd --dashboard to open a browser monitor for the run. It boots a launcher screen where you pick the config, then streams reasoning, live views, and the action timeline. Use --dashboard-language zh-cn for the Chinese UI.
rpent --env libero --dashboard --dashboard-language zh-cn \
--suite libero_goal_task --task 1 --seed 0 --planner claude_codeFor more detailed documentation, see the RPent documentation.
| Flag | Default | Description |
|---|---|---|
--env | — (required) | Environment backend. Currently libero. |
--suite | — (required) | Task suite, e.g. libero_object_task, libero_spatial_swap |
--task | — (required) | Task id within the suite |
--seed | 0 | Random seed |
--planner | api | Reasoning brain: api | claude_code | codex |
--model | — | Model id; for api, prefix the provider (anthropic:…, openai:…, openai-chat:…) |
--max-turns | 100 | Max agent turns |
--max-tokens | 8192 | Max tokens per LLM reply |
--no-images | off | Text-only mode: never send image bytes (for models that reject image input) |
--max-episode-steps | 10000 | Max env steps |
--libero-type | LIBERO_TYPE or pro | LIBERO variant: standard | pro | plus |
--cuda-device | inherited | GPU device(s) exposed to the env / VLA / SAM3 servers |
--dashboard | off | Start the local dashboard for this run |
--dashboard-language | en | Dashboard UI language: en | zh-cn |
--env-endpoint | — (spawn) | [protocol://]host:port of an existing env_server (protocol=http|socket, default http). If unset, one is spawned locally. |
--vla-endpoint | — (spawn) | [protocol://]host:port of an existing vla_server (same rules). If unset, one is spawned locally. |
--sam3-endpoint | — (spawn) | [protocol://]host:port of an existing RPent SAM3 service (same rules). If unset, one is spawned locally. |
If you find RPent or Harness VLA helpful, please cite the paper:
@article{zhang2026harnessvla,
title={Harness VLA: Steering Frozen VLAs into Reliable Manipulation Primitives via Memory-Guided Agents},
author={Zhang, Yixian and Zhang, Huanming and Gao, Feng and Li, Xiao and Liu, Zhihao and Zhu, Chunyang and Qiu, Jiaxing and Yan, Yuchen and Liu, Jiyuan and Tang, Wenhao and Fang, Zhengru and Nie, Yi and Wei, Changxu and Wang, Yu and Ding, Wenbo and Yu, Chao},
journal={arXiv preprint arXiv:2607.08448},
year={2026},
url={https://arxiv.org/abs/2607.08448}
}RPent builds on the simulators, VLA models, and training infrastructure of RLinf, and on the agent SDKs of the broader open-source community — pydantic-ai, the Claude Agent SDK, and the OpenAI Codex SDK. Thanks to the teams behind LIBERO, RoboCasa, robosuite, MuJoCo, and openpi.