Reproduce “Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mijin Yoo, In Cho, Subin Jeon, Jiwoo Lee, Eunbyung Park, Seon Joo Kim
Instok3D predicts object-level groups in 3D scenes from unposed multi-view images in a forward pass. Instance tokens provide identity, while anchor tokens encode geometry and appearance and decode into Gaussians. It uses 2D instance masks, but no 3D annotations. On ScanNet target-view masks with 8 context views, it scores 0.235 AP versus ObjectGS’s 0.178, while ObjectGS performs better on reconstruction metrics.
Key insights
- Directly establishing object instances as the fundamental unit of 3D representation simplifies object-level reasoning and interaction by eliminating the need for post-hoc structure inference.
- A decomposed token structure, separating instance-level identity from local geometric details, facilitates efficient and compact semantic feature distillation while preserving fine-grained information.
- Joint training with simultaneous 2D reconstruction and instance segmentation supervision, including a segmentation loss warm-up, is critical for learning coherent object boundaries and geometry in a feed-forward manner.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.