Reproduce “REACT3D: Recovering Articulations for Interactive Physical 3D Scenes” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhao Huang, Boyang Sun, Alexandros Delitzas, Jiaqi Chen, Marc Pollefeys
REACT3D, developed by researchers at ETH Zurich, is a zero-shot framework that converts static 3D scenes into interactive digital twins by automatically identifying articulated objects, estimating their kinematic parameters, and generating hidden geometry. The system achieves an F1 Score of 0.511 for openable object detection and 91.5% joint-type accuracy, enabling successful real-world robotic manipulation.
Key insights
- The effective integration of open-vocabulary vision foundation models (Grounded SAM, LLaVA, RAM++) allows for zero-shot detection and segmentation of diverse interactive objects without domain-specific training data.
- A multi-view fusion strategy combined with geometric refinement based on OBBs and RANSAC significantly improves the precision of articulation parameter estimation from noisy single-state observations.
- Systematic generation of hidden geometry and meticulous scene assembly (including carving the static background and deduplication) ensures that the output is physically plausible and simulation-ready.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.