Reproduce “Unifying Scene Representation and Hand-Eye Calibration with 3D Foundation Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Weiming Zhi, Haozhan Tang, Tianyi Zhang, Matthew Johnson-Roberson
A unified approach called Joint Calibration and Representation (JCR) performs simultaneous hand-eye calibration and dense 3D scene reconstruction using only RGB images and a 3D foundation model. This method eliminates the need for external markers and produces physically accurate representations with as few as 10-15 input images.
Key insights
- 3D foundation models can provide rich geometric information, including dense point maps and relative camera poses, from a few RGB images, enabling markerless calibration and reconstruction.
- A two-stage calibration process effectively recovers both rotational and translational components of the hand-eye transformation, along with the physical scale factor, from the foundation model's unscaled outputs.
- Implicit neural representations, trained on the transformed 3D points, can efficiently encode continuous scene properties like occupancy, segmentation, and color, creating dense and high-fidelity 3D models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.