Open-source implementation for paper: Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation Main project website is at https://srl-ethz.github.io/rapid-dexterous-writing/
Some elements from the original controller were simplified / modified for the MuJoCo implementation:
- Diagonal covariance matrix
$diag(\mathbf p)$ is a constant scalar$P$ - Estimator regressor uses measured MuJoCo joint velocity instead of delta of commanded velocity
- since we can get ground-truth noise-less joint data in simulation
- Initial excitation is random noise applied to predefined joint groups
- difficult to set predefined postures to interpolate in simulation, where the hand-pen system can't be moved by (human) hand, so this random excitation method was applied
- Task-space command is P + feedforward velocity (I and D terms removed)
- also no EMA filtering for the control output
- Task space controls all three dimensions, not just
XY - 500 Hz simulation, 50 Hz controller
# make venv and activate it
python -m venv venv
source venv/bin/activate
pip install -r requirements.txtDraws a circle as a demo of the Jacobian dexterity controller.
The main code (sim.py) contains all code for the Shadow Hand demo and is kept deliberately minimal (< 400 loc), both as a demonstration of the controller's simplicity and to improve readability and extensibility.
# for Shadow Hand sample
python sim.py
# for Wuji Hand 2 sample
python sim_wuji_hand_2.pySteps to make a scene in which the hand holds a pen, defined by a keyframe. The keyframe can be activated in the MuJoCo GUI with the Key slider in the upper right.
- prepare your high-quality MJCF robot hand model which includes position actuators
- make a tilted version of the model where the hand pose is set to a "writing" pose
- e.g.
wuji_hand_2/right_tilted.xml - unfortunately the
<attach>feature cannot be used to load the model for our case, since it had some trouble with settingqposkeyframe fields for the attached model
- e.g.
- create a new MJCF file which includes the tilted robot hand model and the pen body
- e.g.
wuji_hand_2/scene_pen.xml - uses the
<include>feature to add the tilted model to the scene, which doesn't have the issue with keyframes - you can copy the
<body name="pen" ...from that file for the pen model
- e.g.
- set good-looking "writing" hand pose manually by adjusting the Control sliders in the MuJoCo GUI (ignore the pen for now)
- once you're satisfied, "Copy state" from the GUI and fill in the
ctrlfields for your keyframe
- once you're satisfied, "Copy state" from the GUI and fill in the
- move the pen into the hand- the pen can be moved kinematically by pausing, double clicking the pen, and dragging it while pressing ctrl (right drag to rotate, left drag to move)
- adjust the pen's grip in the hand (by going back and forth between modifying the Control sliders and pausing & moving the pen kinematically)
- "Copy Pose" again and paste both the
ctrlandqpodfields to your keyframe (don't copy thetimefield, the time is used to detect resets in the code), so that the robot and pen take the correct pose the moment the keyframe is loaded.
- make the sim script for your robot - use
sim_wuji_hand_2.pyas reference. - you may have try around a lot to adjust parameters like the robot & pen pose and control params until you get a combination that works- be patient!
- robot hands with wrist DoFs (like the Shadow Hand) should generally have better performance for writing