[Paper]

A curated list of papers and resources on Reinforcement Learning of Vision-Language-Action (RL-VLA) models for Robotic Manipulation. This repository provides a comprehensive overview of training paradigms, methodologies, and state-of-the-art approaches in RL-VLA research.

🔥 [November 2025] Our comprehensive survey paper "A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation" is now available on TechRxiv! Stay tuned for future updates.

• Awesome RL-VLA for Robotic Manipulation 🤖
  • 📢 Latest News
  • 📖 Table of Contents
  • 🔍 Overview
  • 🚀 Training Paradigms
    • Offline RL-VLA
    • Online RL-VLA
    • Test-time RL-VLA
  • 📚 Paper Collection
    • Legend
    • Offline RL-VLA
    • Online RL-VLA
    • Offline + Online RL-VLA
    • Test-time RL-VLA
  • 🔗 Useful Resources
    • 🎯 RL-VLA Action Optimization
    • Base VLA Models
    • Datasets & Benchmarks
    • Frameworks & Tools
  • 🤝 Contributing
    • Contribution Guidelines
  • 📄 Citation
  • ⭐ Star History

RL training is crucial for enabling VLAs to generalize out-of-distribution (OOD) from large-scale pre-trained data. Existing RL-VLA training paradigms can be categorized into three types based on how agents obtain and utilize feedback from the environment:

• Online RL-VLA: Direct interaction with the environment during training
• Offline RL-VLA: Learning from static datasets without further environmental interaction
• Test-time RL-VLA: Models adapt their behavior during deployment without altering parameters

Offline RL trains VLA models on pre-collected static datasets, enabling learning independently from environment interactions. This paradigm is suitable for high-risk or resource-constrained deployment scenarios.

Key Research Directions:

• Data Utilization: Effective utilization of static datasets for policy improvement
• Objective Modification: Customizing RL objectives for novel architectures and data augmentation

Online RL-VLA enables interactive policy learning through continuous environment interaction, empowering pre-trained VLAs with adaptive closed-loop control capability for real-world OOD environments.

Key Research Directions:

• Policy Optimization: Direct policy improvement based on environmental rewards
• Sample Efficiency: Learning effective policies with limited interaction budget
• Active Exploration: Efficient exploration strategies for higher performance gains
• Training Stability: Ensuring consistent policy updates and convergence
• Infrastructure: Scalable frameworks for online RL-VLA training

Test-time RL-VLA adapts behavior during deployment through lightweight updates, addressing the expensive cost of full model fine-tuning in real-world scenarios.

Key Adaptation Mechanisms:

• Value Guidance: Using pre-trained value functions to influence action selection
• Memory Buffer Guidance: Retrieving relevant historical experiences during inference
• Planning-guided Adaptation: Explicit reasoning over future action sequences

• Action: AR (Autoregressive), Diffusion, Flow (Flow-matching)
• Reward: D (Dense Reward), S (Sparse Reward)
• Model Type: MB (Model-based), MF (Model-free)
• Environment: Sim. (Simulation), Real (Real-world)
• Task: MT (Multi-task), ST (Single-task)
• Policy: On-Policy, Off-Policy, Hybrid (mixed on/off-policy), Test-time (inference-time adaptation)

Note: The 🔗 symbol in the Project column indicates papers with available project pages, GitHub repositories, or demo websites.

Different VLA architectures require distinct RL optimization strategies based on their action generation mechanisms:

🔤 Autoregressive VLA: Optimizes actions at the token-level. Each action token is individually optimized through RL, enabling fine-grained control over action sequences but requiring careful handling of sequential dependencies. 🌊 Generative VLA (Diffusion/Flow): Optimizes along the action generation process at the sequence-level. The entire action trajectory is optimized as a cohesive unit through the denoising or flow-matching process, providing holistic action optimization. 🔗 Dual-system VLA: Optimizes at the bridge-level. RL decides which high-level action proposal to pass to the fast controller, creating a hierarchical optimization approach that complements both token-level and sequence-level methods.

• GR00T-N1 - NVIDIA series
• π0 - PI series
• OpenVLA - Open-source VLA model
• Octo - Generalist robot policy
• RT-1 - Robotics Transformer

• Open X-Embodiment - Large-scale robotic datasets
• LIBERO - Benchmark for lifelong robot learning
• SimplerEnv - Benchmark for real-sim robot learning
• RoboTwin - Benchmark for bimanual robot learning
• DeepPHY - Benchmark for physical reasoning

• RLinf - Infrastructure for online RL fine-tuning of VLAs
• RLinfv0.2 - Infrastructure for real world RL

We welcome contributions to this awesome list! Please feel free to:

1. Add new papers: Submit a PR with new RL-VLA papers following the existing format
2. Update information: Correct any errors or update paper information
3. Suggest improvements: Propose better organization or additional sections

• Ensure papers are relevant to RL-VLA research
• Include paper links, project pages (if available), and key details
• Follow the existing table format for consistency
• Add a brief description for new paradigms or significant methodological contributions

If you find this repository useful, please consider citing:

@article{pine2025rlvla,
  title={A Survey on Reinforcement Learning of Vision-Language-Action Models for Robotic Manipulation},
  author={Haoyuan Deng, Zhenyu Wu, Haichao Liu, Wenkai Guo, Yuquan Xue, Ziyu Shan, Chuanrui Zhang, Bofang Jia, Yuan Ling, Guanxing Lu, and Ziwei Wang},
  journal={TechRxiv},
  year={2025},
  doi={10.36227/techrxiv.176531955.54563920/v1},
  note={Preprint}
}

Star this repository if you find it helpful!