RPent is an physical-agent framework that puts a large language model in the loop as the decision-making brain. The LLM does high-level reasoning and calls tools; a Vision-Language-Action (VLA) policy such as Pi0.5 or RLDX-1 executes the low-level motor actions; a simulator (LIBERO or RoboCasa) closes the loop by returning observations and rendered frames. Reasoning, action, and simulation each run in their own process, so heavyweight GPU models and the physics engine never fight over one Python interpreter.

• LLM-in-the-loop control. The LLM is not fine-tuned — it drives the robot purely by calling tools (pi0_pick, move_to, rotate_wrist, back_project, finish, …). Each tool result is fed back as multimodal context (text + rendered images), so the model reasons over what it actually sees.
• Three-process architecture. The agent process (LLM cerebrum + toolkit, no torch), the env_server (simulator + EGL rendering), and the vla_server (GPU policy weights) are separate processes wired by lightweight RPC. Either heavyweight process can be restarted, moved to another GPU, or pointed at a remote host independently.
• Pluggable reasoning brains (cerebrums). Swap the decision brain with one flag — --cerebrum {api, claude_code, codex} — without touching the tools or prompts:
  • api — a provider-agnostic tool-calling loop built on pydantic-ai (Anthropic / OpenAI / OpenAI-compatible), with prompt caching and history-image pruning.
  • claude_code — the Claude Agent SDK, exposing the toolkit as an in-process MCP server.
  • codex — the OpenAI Codex SDK, bridged to the toolkit over an HTTP MCP server.
• Two environments, two VLAs, one contract. LIBERO (Pi0.5 over HTTP) and RoboCasa (RLDX-1 over socket-RPC) share the exact same env/vla process split; only the wire codec differs, chosen to fit each env's observation shape.
• Live dashboard. An optional --dashboard starts a local FastAPI monitor that streams the agent's reasoning, real-time camera / Pi0 views, an action timeline, and clip replays — with a bilingual UI (--dashboard_language {en, zh-cn}).
• Add an environment by dropping a package on disk. No central registry to edit — see Adding a new environment.

A single run is an LLM-in-the-loop cycle:

1. The LLM reasons about the task and calls a tool (e.g. pi0_pick).
2. The tool's primitive driver asks the vla_server for an action chunk (predict / vla_infer).
3. The env_server executes that chunk (chunk_step for LIBERO, stepwise step for RoboCasa).
4. The env renders the resulting observation and camera frames.
5. Results are turned into text + image content blocks and fed back to the LLM for the next turn.

The loop ends when the LLM calls the finish tool (success / failure / stuck) or hits --max_turns / --max_episode_steps.

RPent runs on top of a forked branch of RLinf for the simulators and VLA models. Clone them side by side.

1. Clone RLinf and RPent side by side.

mkdir workspace && cd workspace
# RPent depends on a forked branch of RLinf; it will be merged back to main after more iterations.
git clone https://github.com/jx-qiu/RLinf -b feature/physicalagent rlinf
git clone https://github.com/RLinf/RPent rpent

2. In RLinf, create an openpi + LIBERO virtualenv.

cd rlinf
bash requirements/install.sh embodied --env libero --model openpi --use-mirror --venv ../.venv-opi-libero
cd ..
source .venv-opi-libero/bin/activate

3. Install RPent's extra dependencies on top of that venv.

cd rpent
uv sync --active --inexact
bash scripts/install_libero_pro_plus.sh

4. Configure keys and checkpoints, then run.

# LLM API keys (the `api` cerebrum)
export ANTHROPIC_BASE_URL=https://xxx
export ANTHROPIC_API_KEY=sk-xxx
export OPENAI_BASE_URL=https://xxx
export OPENAI_API_KEY=sk-xxx

# VLA checkpoint — download from
# https://huggingface.co/datasets/RLinf/rlinf-pi05-libero-130-fullshot-sft
export PI05_CHECKPOINT_PATH=/path/to/rlinf-pi05-libero-130-fullshot-sft
export LIBERO_TYPE=pro
export CUDA_DEVICE=0

# Run one task: libero_object_swap, task 2, seed 0, using the `api` cerebrum
# with an Anthropic model and an 8192-token cap.
#   • OpenAI-compatible chat endpoints:  --model openai-chat:glm-5.2
#   • OpenAI responses endpoints:        --model openai:gpt-5.5
#   • claude_code / codex cerebrums:     no provider prefix, e.g. --model claude-opus-4-8
python cli/main.py --suite libero_object_swap --task 2 --seed 0 \
  --cerebrum api --model anthropic:claude-opus-4-8 --max_tokens 8192

Add --dashboard to open a browser monitor for the run. It boots a launcher screen where you pick the config, then streams reasoning, live views, and the action timeline. Use --dashboard_language zh-cn for the Chinese UI.

python cli/main.py --dashboard --dashboard_language zh-cn \
  --suite libero_goal_task --task 1 --seed 0 --cerebrum claude_code

RoboCasa uses a separate entrypoint and setup guide.

bash scripts/setup_robocasa.sh                                # one-time setup
bash scripts/run_robocasa.sh PickPlaceCounterToCabinet 0 0    # <task> <gpu> <seed>

See SETUP_ROBOCASA.zh.md for the full RoboCasa365 + RLDX-1 walkthrough.

• Adding a new environment — plug a new simulator / robot into the runner (中文).
• RoboCasa setup — RoboCasa365 + RLDX-1 install and run guide.
• docs/ — the full documentation index.

RPent builds on the simulators, VLA models, and training infrastructure of RLinf, and on the agent SDKs of the broader open-source community — pydantic-ai, the Claude Agent SDK, and the OpenAI Codex SDK. Thanks to the teams behind LIBERO, RoboCasa, robosuite, MuJoCo, and openpi.