Robust 6-DoF perception for autonomous modular robot assembly
An end-to-end Physical AI pipeline — RGB-D perception, neural segmentation, model-based 6-DoF pose estimation, temporal tracking and robotic manipulation — for precision modular assembly, validated in the lab and on a JAXA lunar field testbed.
- Synthetic images
- ~30,000
- Pose
- 6-DoF + tracking
- Validation
- Lab + JAXA field
- Challenge
- Precision assembly — grasping, alignment and connector insertion — depends on full 6-DoF pose, and frame-wise pose estimation becomes unstable under occlusion, fast motion and temporary loss of visibility.
- Approach
- RGB-D sensing → SAM2 segmentation → FoundationPose 6-DoF estimation → temporal tracking with persistence and re-initialisation → transform-frame alignment → a state-machine assembly sequence, designed for failure recovery rather than only nominal execution.
- Outcome
- A single perception-to-manipulation workflow that completes modular assembly through occlusion and motion, verified physically in the lab and at a JAXA lunar field.
Stack
- FoundationPose
- SAM2
- RGB-D
- NVIDIA NVISII
- Unreal Engine (domain randomisation)
- NeRF references
- ROS / TF
- Robot manipulation
Conducted as the founder's Master's research at the Tohoku University Space Robotics Laboratory, with physical validation at a JAXA lunar field testbed (2026).
The engineering challenge
Precision assembly needs more than object detection. Grasping, aligning and inserting a connector depend on both where the target is and how it is oriented — so the system has to estimate translation and rotation, and reason about grasp feasibility and connector angle before contact.
The hard part is that this has to survive real manipulation:
- Occlusion — the gripper and robot links partially block the target while it is being handled.
- Motion — rapid object motion destabilises frame-by-frame pose estimates.
- Visibility — the target briefly leaves the camera view and the pipeline has to recover, not reset.
The perception stack
~30,000 synthetic images were generated with NVIDIA NVISII, with Unreal Engine domain-randomisation work to close sim-to-real gaps in illumination and sensor domain. Synthetic generation exposed the stack to varied viewpoints, backgrounds and scene conditions while making the remaining gaps measurable.
- 3D reference assets — base meshes and NeRF-enhanced textured references were aligned to canonical object frames and used as geometric and appearance priors for pose estimation.
- Segmentation to pose — SAM2 masks isolate the modular components from the RGB stream and constrain correspondence-based 6-DoF estimation with FoundationPose.
Robot integration
Pose estimates feed a state-machine assembly sequence:
- Detect — 6-DoF localisation of the connector and dummy module.
- Grasp — grasp estimation on the target.
- Align — relative transform-frame alignment, with the required offset correction.
- Connect — establish the physical connection.
Failure-aware tracking. Pose persistence and re-initialisation keep the sequence continuous through occlusion, fast motion and temporary loss of visibility — the system is designed for recovery, not just nominal execution.
End to end
The final workflow chains synthetic data generation, 3D object assets, NeRF reconstruction, SAM-based segmentation, object detection, transform-frame definition, alignment and offset correction, connection establishment, and testing in the JAXA lunar field.
Key takeaway
A single perception-to-manipulation workflow enables robust modular robot assembly under challenging conditions — and the same failure-recovery discipline carries directly into industrial deployments where the target is occluded, moving, or briefly out of view.