GaussianFeels
Object-centric Gaussian SLAM for visuo-tactile in-hand manipulation
M.S. Thesis · Soft Robotics & Bionics Lab, Seoul National University
When a robot hand grasps an object, it occludes exactly the region it most needs to perceive. GaussianFeels fuses RGB-D vision, DIGIT tactile contact geometry, and hand proprioception into one explicit object-centric 3D Gaussian Splatting map, reconstructing and tracking objects online, through the occlusion, with no CAD model.
Method & results
- One map, every job: a single object-centric 3D Gaussian state serves training, rendering, frozen-map SDF pose tracking, reconstruction evaluation, and manipulation-facing geometry. It replaces the neural-implicit SDF as the shared representation for online, model-free visuo-tactile object SLAM.
- Pose is recovered by a multi-residual Levenberg-Marquardt optimiser solving SE(3) against a frozen dual-sigma Gaussian-density anchor SDF, fusing synchronized RGB-D, tactile, and proprioceptive observations in one canonical frame.
- A frame-zero branch generates a shape estimate from a single RGB crop with an image-to-3D model, then progressively replaces generated geometry with measured geometry as the episode progresses.
- Evaluated on the FeelSight benchmark (8 simulation + 6 real-robot objects) across its simulation, real-robot, and occlusion splits, against the NeuralFeels and V-HOP baselines.
- Pose mode: ADD-S 1.13 mm in simulation at 50 FPS and 2.42 mm on coverage-adequate real objects at 37 FPS, on a single RTX 5090. Map mode: observed-surface F-score@5 mm of 0.997 (sim) / 0.950 (real).
- Lower pose error than NeuralFeels on every simulation aggregate at 7.0–12.4× the per-frame speed (5.8–8.3× on real); full-surface F-score 0.997 vs 0.908 on the matched object, cutting reconstruction time from ≈52 min to 12 s.
- Edges V-HOP on occlusion-split ADD-S (1.45 vs 1.46 mm) at 48.8 vs 32 FPS, all with no supplied CAD model.
- Developed inside Korea's national “Alchemist” humanoid programme (MOTIE), bringing visuo-tactile SLAM from research prototype to the Phase-2 full-scale humanoid.