Skip to content

feat(manipulation): heuristic grasps after the provider's in the M1 pick - #4523

Draft
mustafab0 wants to merge 5 commits into
mustafa/feat/m1-rescan-posefrom
mustafa/feat/m1-grasp-fallback
Draft

mustafab0 wants to merge 5 commits into
mustafa/feat/m1-rescan-posefrom
mustafa/feat/m1-grasp-fallback

Conversation

@mustafab0

Copy link
Copy Markdown
Contributor
  • The provider's grasps come first (25 tried), then heuristic grasps from the same points.
  • The heuristic becomes a plain function and proposes a few distinct grasps, with widths measured end to end.
  • The pick reports rejections per source.

Eval setup for the numbers below: SceneReplica, 100 cases, xArm7 in sim2 (MuJoCo), scripted pick (no LLM), paired on the same cases. "Perfect perception" means the object's true mesh from the sim (the grasp lab); "camera" means the wrist camera's scene registration.

Part 5 of 13 in the closed-loop grasp (M1) stack; based on mustafa/feat/m1-rescan-pose, so this PR shows only its own change.

HeuristicGraspModule returned one top-down grasp, so a retry after a
miss had nothing different to try. It now returns, in rank order:

- the grasp it gave before (median XY, halfway up, jaws across the
  narrow side), score 1.0, always first;
- for objects at least min_split_length (0.10 m) long, one grasp in the
  middle of each half along the length, at that half's own median XY and
  height, scores 0.8 and 0.7 (the half with more points first);
- the middle grasp turned a quarter turn, score 0.5, when the object's
  length fits in the open jaws.

A half wider than max_grasp_width (0.085 m, the xArm gripper's opening)
across the jaws is not offered, nor is the turned grasp for a longer
object. Both are config fields. Scores only rank the candidates.
…und turns

The width checks used the 5th to 95th percentiles of the points, which
read an evenly spread object about 10% short: a 9 cm block measured
8.1 cm and the turned grasp brought 8.5 cm jaws down on it. Widths are
now measured between the 1st and 99th percentiles, and a grasp is
offered only when the object is narrower than the jaw opening by
jaw_clearance (0.01 m, config). The same holds for each half.

The turned grasp is also skipped when the footprint's length and width
differ by less than 1 cm: on a round or square object it holds the same
way as the first grasp and would only use up a retry.
Move HeuristicGraspModule's grasp math into heuristic_grasps(cloud,
config), with the grasp rules in HeuristicGraspConfig. The module's
propose_grasps calls it and returns the same grasps as before, so other
modules can compute heuristic grasps without a second grasp provider.
…M1 pick

Each grasp try now plans to the provider's best max_grasp_attempts new
candidates, then to heuristic grasps computed from the same points. The
re-scan after a miss proposes both again, and a candidate from either
source that repeats a grasp already tried is skipped. A GraspGenX pick
then still has grasps when GraspGenX returns none, raises, or its
grasps cannot be reached or miss within the retry budget.

- A provider error is logged and treated as no provider grasps; the try
  records it as grasp_provider_error.
- Heuristic grasps within retry_skip_distance and retry_skip_angle of a
  provider grasp being tried are dropped; a provider that is
  HeuristicGraspModule gets no second copy.
- Every pick result carries grasp_sources; each try records its source
  and candidates_by_source.
- PickAndPlaceModuleConfig.heuristic_fallback (default True) turns it
  off, giving the provider's grasps only and raising its errors as
  before. heuristic_grasp sets the gripper fit.
- _repeats uses the shared same_grasp check.
…e M1 pick, report rejections

GraspGenX's best-scoring xArm grasps on SceneReplica objects are mostly side
grasps at table height, many approaching from below or from the far side.
The planner's Pink IK rejects those in about 0.25-0.3 s each (10 restarts of
200 iterations), so with only 5 tried each try nearly always fell through to
the heuristic grasp. max_grasp_attempts now defaults to 25.

Each record in tries now carries rejected_by_source (how many grasps from each
source the planner found no path to in that try), and the pick result carries
the total over the whole pick, including a final try that reached no grasp. A
move to a grasp that stops part-way is now recorded as a try too, so its
source and rank are reported.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go2 replay realtime (arm64)

Details
Benchmark suite Current: df052cf Previous: 1e7b491 Ratio
peak memory 1619.41 MB 1569.52 MB 1.03
peak threads 356 threads 352 threads 1.01
network (transport) 2724.744 MB 2722.016 MB 1.00
disk write 1.469 MB 1.492 MB 0.98
instructions 115.53 G 115.634 G 1.00

This comment was automatically generated by workflow using github-action-benchmark.

@codecov

codecov Bot commented Oct 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.75904% with 1 line in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
dimos/manipulation/pick_and_place_module.py 98.24% 1 Missing ⚠️
@@                       Coverage Diff                       @@
##           mustafa/feat/m1-rescan-pose    #4523      +/-   ##
===============================================================
+ Coverage                        80.62%   80.67%   +0.04%     
===============================================================
  Files                             1655     1657       +2     
  Lines                           156189   156540     +351     
  Branches                         13170    13185      +15     
===============================================================
+ Hits                            125931   126289     +358     
+ Misses                           26689    26683       -6     
+ Partials                          3569     3568       -1     
Components Coverage Δ
Tests 95.37% <100.00%> (+0.01%) ⬆️
Flag Coverage Δ
OS-macos-latest 75.00% <99.75%> (+0.06%) ⬆️
OS-ubuntu-24.04-arm 74.75% <99.75%> (+0.05%) ⬆️
OS-ubuntu-latest 75.11% <99.75%> (+0.06%) ⬆️
Py-3.10 75.11% <99.75%> (+0.05%) ⬆️
Py-3.11 75.10% <99.75%> (+0.06%) ⬆️
Py-3.12 75.11% <99.75%> (+0.06%) ⬆️
SelfHosted-Large 32.10% <27.22%> (-0.01%) ⬇️
SelfHosted-Linux 39.96% <27.22%> (-0.02%) ⬇️
SelfHosted-macOS 40.24% <27.22%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
dimos/manipulation/grasping/grasp_fallback.py 100.00% <100.00%> (ø)
dimos/manipulation/grasping/heuristic_grasp.py 100.00% <100.00%> (+4.87%) ⬆️
dimos/manipulation/grasping/test_grasp_fallback.py 100.00% <100.00%> (ø)
...imos/manipulation/grasping/test_heuristic_grasp.py 100.00% <100.00%> (ø)
dimos/manipulation/test_pick_and_place_unit.py 100.00% <100.00%> (ø)
dimos/manipulation/pick_and_place_module.py 92.85% <98.24%> (+0.34%) ⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@mustafab0
mustafab0 added this pull request to stack #4532 October 11, 2026 06:49

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant