Reproduce “Pandora: Articulated 3D Scene Graphs from Egocentric Vision” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Alan Yu, Yun Chang, Christopher Xie, Luca Carlone
Pandora builds an articulated 3D scene graph from a person’s interaction recording. It links objects and distinguishes items that move with articulated parts from items stored behind them. In simulated Kitchen scenes, its relationship F1 was 100.0, versus 41.2 without articulation geometry; in each real-world kitchen, it correctly identified and associated 5/6 concealed objects with a parent and semantic label.
Key insights
- Egocentric data, reflecting natural human interaction, provides a more comprehensive signal for discovering and modeling object articulation compared to limited robot-centric observations.
- Integrating explicit articulation models (joint type, axis, pivot, state) into 3D scene graphs substantially improves the representation of dynamic objects and their relationships, like containment.
- A method combining hand pose trajectories during interaction with geometric constraints enables accurate estimation of articulation parameters and their dynamic states.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.