Building Cooperative Embodied Agents Modularly with Large Language Models

This repo contains codes for the following paper:

Hongxin Zhang*, Weihua Du*, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, Chuang Gan: Building Cooperative Embodied Agents Modularly with Large Language Models

Paper: Arxiv

Project Website: Co-LLM-Agents

[8/1/2023]: We update the VirtualHome Simulator executable we used here. If you met XDG_RUNTIME_DIR not set in the environment error previously, please check if you are using the new version we provided.

For detailed instructions on the installation of the two embodied multi-agent environments Communicative Watch-And-Help and ThreeDWorld Multi-Agent Transport, please refer to the Setup sections in envs/cwah/README.md and envs/tdw_mat/README.md respectively.

Step 1: Get the VirtualHome Simulator and API

Clone the VirtualHome API repository under envs:

cd envs
git clone --branch wah https://github.com/xavierpuigf/virtualhome.git

Download the Simulator (Linux x86-64 version), and unzip it in envs.

The files should be organized as follows:

|--cwah/
|--virtualhome/
|--executable/

Step 2: Install Requirements

cd cwah
conda create --name cwah python=3.8
conda activate cwah
pip install -r requirements.txt

Run the following commands step by step to setup the environments:

cd envs/tdw_mat
conda create -n tdw_mat python=3.9
conda activate tdw_mat
pip3 install -e .

After that, you can run the demo scene to verify your setup:

python demo/demo_scene.py

The main implementation code of our Cooperative LLM Agents is in envs/*/LLM and envs/*/lm_agent.py.

We also prepare example scripts to run experiments with HP baseline and our Cooperative LLM Agent under the folder envs/*/scripts.

For example, to run experiments with two LLM Agents on Communicative Watch-And-Help, run the following command in folder envs/cwah.

./scripts/symbolic_obs_llm_llm.sh

We noticed many interesting agents' behaviors exhibited in our experiments and identified several cooperative behaviors.

There are more interesting cases and demos on our website!

If you find our work useful, please consider citing:

@article{zhang2023building,
  title={Building Cooperative Embodied Agents Modularly with Large Language Models},
  author={Zhang, Hongxin and Du, Weihua and Shan, Jiaming and Zhou, Qinhong and Du, Yilun and Tenenbaum, Joshua B and Shu, Tianmin and Gan, Chuang},
  journal={arXiv preprint arXiv:2307.02485},
  year={2023}
}