[Paper] | [Website]

• 2026-06-17 — Accepted by IROS 2026 🎉
• 2026-06-07 — 🥈 2nd place at the NeuroDesign Competition in ICRA 2026 NeuroDesign in HRI Workshop

conda create -n tasc python=3.8
conda activate tasc
pip install -r requirements.txt
pip install -e .

Note: These installations are optional. We provide offline segmentation and grasp pose results for quick testing. Install this if you want to try the complete pipeline.

• Grounded-SAM

  Clone the Grounded-SAM repository, set some environment variables and install GroundingDINO and SAM.

  git clone git@github.com:IDEA-Research/Grounded-Segment-Anything.git
  export AM_I_DOCKER=False
  export BUILD_WITH_CUDA=True
  export CUDA_HOME=/path/to/cuda/
  python -m pip install -e segment_anything
  pip install --no-build-isolation -e GroundingDINO

  Download DINO and SAM checkpoints, in TASC root directory:

  mkdir ckpts && cd ckpts
  wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
  wget https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth

• GraspNet

1. Clone the repo.

   cd ~/tasc
   git clone https://github.com/graspnet/graspnet-baseline.git
   cd graspnet-baseline

2. Install packages, pointnet2 operators, knn operator and graspnetAPI.

   pip install -r requirements.txt
   cd pointnet2
   python setup.py install
   cd ../knn
   python setup.py install
   cd ..
   git clone https://github.com/graspnet/graspnetAPI.git
   cd graspnetAPI
   pip install .

3. Download GraspNet checkpoints: checkpoint-rs.tar, in TASC root directory:

   mv ./checkpoint-rs.tar ~/tasc/ckpts

   Please refer to graspnet-baseline repo for troubleshooting.

Run TASC with stored query results:

python tasc/tasc_simulation.py 

Franka Panda Teleoperation with keyboard:

Keys                            Command
Ctrl+q                          reset simulation
spacebar                        toggle gripper (open/close)
up-right-down-left              move horizontally in x-y plane
.-;                             move vertically
o-p                             rotate (yaw)
y-h                             rotate (pitch)
e-r                             rotate (roll)

Other arguments:

# Run TASC with debug information
python tasc/tasc_simulation.py --debug

# Run TASC with online query
python tasc/tasc_simulation.py --use_llm

# Run TASC with grasping planner
python tasc/tasc_simulation.py --use_graspnet

# Run in pure teleoperation mode (for comparison)
python tasc/tasc_simulation.py --pure_teleop

# Record operation data
python tasc/tasc_simulation.py --record

Analyse operation data:

python tasc/result_analyzer.py --input tasc --episode_idx 1 --task_idx 1

If you need to run TASC on a headless server (no monitor attached) and display the GUI on your local machine, you can use X11 forwarding.

1. Install XLaunch or Xming.
2. Start XLaunch (keep default settings).
3. In PowerShell, set the display environment variable:

$env:DISPLAY="localhost:0.0"

4. Connect to the server with X11 forwarding enabled:

ssh -Y <user>@<server-ip>

1. Make sure you have an X server installed locally (Linux usually has it by default; macOS can use XQuartz).
2. Connect to the server with X11 forwarding:

ssh -Y <user>@<server-ip>

See our real-world setup in this repo.

In shrot, we use moveit_servo to teleoperate a Franka Research3 robot in ROS2 environment. We also seperate the ROS2 and CUDA environment and use websocket to communicate, to aviod python version conflict.

We gratefully acknowledge the following open-source projects that greatly facilitated our work:

• Segmentation pipeline adapted from MOKA.
• Goal prediction adapted from Improved APF.

We also thank the robosuite community for providing an excellent simulation framework that supported our research.

If you find our paper interesting or useful in your work, please cite our paper:

@misc{fu2025tasc,
      title={TASC: Task-Aware Shared Control for Teleoperated Manipulation}, 
      author={Ze Fu and Pinhao Song and Yutong Hu and Renaud Detry},
      year={2025},
      eprint={2509.10416},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2509.10416}, 
}