Skip to content

feat(manipulation): GraspGenX drops grasps whose gripper would hit the scene - #4524

Draft
mustafab0 wants to merge 2 commits into
mustafa/feat/m1-grasp-fallbackfrom
mustafa/feat/graspgenx-scene-filter
Draft

mustafab0 wants to merge 2 commits into
mustafa/feat/m1-grasp-fallbackfrom
mustafa/feat/graspgenx-scene-filter

Conversation

@mustafab0

@mustafab0 mustafab0 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GraspGenX can check its grasps against the scene cloud with the xArm gripper as boxes (grasp_scene_filter). The xArm GraspGenX gripper geometry moves to its own module.

Later in the stack (#4529) this box filter is switched off by default in favour of the accurate contact check.

Eval setup for the numbers below: SceneReplica, 100 cases, xArm7 in sim2 (MuJoCo), scripted pick (no LLM), paired on the same cases. "Perfect perception" means the object's true mesh from the sim (the grasp lab); "camera" means the wrist camera's scene registration.

Part 6 of 13 in the closed-loop grasp (M1) stack; based on mustafa/feat/m1-grasp-fallback, so this PR shows only its own change.

…wn module

XARM_GRIPPER_SWEEP_VOLUME and XARM_GRASP_FRAME_TO_TCP move from
xarm/blueprints/grasp.py to xarm/grasp_geometry.py, a module with no imports,
so the sim2 blueprints can use them without building the xarm-grasp stack.
… scene

GraspGenX sees only the object's points, so it proposes grasps from under or
through the table. Nothing checked the gripper at the grasp pose itself: the
planner checks the path to the pre-grasp pose, the last straight move runs with
collision checking off, and the voxel-map ignore box blanks the table under the
target.

GraspGenXModule gets propose_grasps_in_scene(object, scene, threshold). It runs
the model as propose_grasps does, then GraspGenX's own filter_colliding_grasps
on the model's grasp-frame poses, and returns the clear grasps best first with
how many it removed (SceneGrasps). propose_grasps is unchanged.

The gripper shape is a new GraspGenXConfig.gripper_body: boxes in the sweep
volume's frame. The sweep-volume runtime only has a 1 cm placeholder mesh, and
the sweep volume is just the gap between the jaws, so XARM_GRIPPER_BODY gives
the xArm palm (as wide as its flange), knuckles and fingers, measured from the
URDF meshes at drive_joint = 0. Points 7 mm apart on the box faces go to the
filter; the distance batch is sized to stay under 256 MB.

pick_object asks for the check when grasp_scene_filter is on (default) and the
provider offers it (optional SceneGraspGenSpec ref). The scene is the scene
registration's camera cloud of the latest scan with the target masked out,
cut to 0.15 m around the target's points, minus points within 1 cm of the
target's points and within 0.15 m of a tool tip (the wrist camera sees its own
fingers). The table stays. scene_collision_threshold defaults to 1 cm, the
scene cloud's cell size. Each try and the pick result record
scene_filtered_by_source next to rejected_by_source, summed over retries, and
scene_filter_skipped when no scene cloud was available. Heuristic grasps are
not checked.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: 16fb2d9 Previous: 1e7b491 Ratio
peak memory 1601.016 MB 1569.52 MB 1.02
peak threads 344 threads 352 threads 0.98
network (transport) 2721.694 MB 2722.016 MB 1.00
disk write 1.418 MB 1.492 MB 0.95
instructions 115.706 G 115.634 G 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.38710% with 5 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/manipulation/pick_and_place_module.py 92.45% 3 Missing and 1 partial ⚠️
dimos/manipulation/grasping/grasp_gen_x/module.py 96.00% 1 Missing ⚠️
@@                        Coverage Diff                         @@
##           mustafa/feat/m1-grasp-fallback    #4524      +/-   ##
==================================================================
+ Coverage                           80.67%   80.75%   +0.08%     
==================================================================
  Files                                1657     1659       +2     
  Lines                              156540   156831     +291     
  Branches                            13185    13200      +15     
==================================================================
+ Hits                               126289   126654     +365     
  Misses                              26683    26683              
+ Partials                             3568     3494      -74     
Components Coverage Δ
Tests 95.38% <100.00%> (+0.01%) ⬆️
Flag Coverage Δ
OS-macos-latest 75.04% <98.38%> (+0.03%) ⬆️
OS-ubuntu-24.04-arm 74.80% <98.38%> (+0.04%) ⬆️
OS-ubuntu-latest 75.16% <98.38%> (+0.04%) ⬆️
Py-3.10 75.15% <98.38%> (+0.04%) ⬆️
Py-3.11 75.14% <98.38%> (+0.04%) ⬆️
Py-3.12 75.15% <98.38%> (+0.04%) ⬆️
SelfHosted-Large 32.09% <26.77%> (-0.01%) ⬇️
SelfHosted-Linux 39.93% <26.77%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/manipulation/grasping/grasp_fallback.py 100.00% <100.00%> (ø)
dimos/manipulation/grasping/grasp_gen_spec.py 100.00% <100.00%> (ø)
...s/manipulation/grasping/grasp_gen_x/test_module.py 100.00% <100.00%> (ø)
dimos/manipulation/grasping/test_grasp_fallback.py 100.00% <100.00%> (ø)
dimos/manipulation/test_pick_and_place_unit.py 100.00% <100.00%> (ø)
dimos/robot/manipulators/xarm/blueprints/grasp.py 91.66% <100.00%> (+5.95%) ⬆️
dimos/robot/manipulators/xarm/grasp_geometry.py 100.00% <100.00%> (ø)
...mos/robot/manipulators/xarm/test_grasp_geometry.py 100.00% <100.00%> (ø)
dimos/manipulation/grasping/grasp_gen_x/module.py 95.94% <96.00%> (+5.74%) ⬆️
dimos/manipulation/pick_and_place_module.py 95.39% <92.45%> (+2.53%) ⬆️

... and 5 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mustafab0
mustafab0 added this pull request to stack #4532 October 11, 2026 06:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant