Reproduce “Leverage Cross-Attention for End-to-End Open-Vocabulary Panoptic Reconstruction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xuan Yu, Yuxuan Xie, Yili Liu, Haojian Lu, Rong Xiong, Yiyi Liao, Yue Wang
PanopticRecon++ introduces an end-to-end framework for open-vocabulary panoptic reconstruction, integrating 3D geometry, appearance, semantics, and instance information by leveraging 3D Gaussian-modulated instance tokens and a cross-attention mechanism. The method achieves competitive performance, often leading in Panoptic Quality and Semantic Quality on Replica, ScanNet-V2, and ScanNet++, and reconstructs geometry with accuracy under 1cm on Replica and ScanNet++.
Key insights
- Modeling 3D instances with 3D Gaussian-modulated tokens effectively integrates explicit spatial priors, enabling more robust instance representation and disambiguation, especially when objects do not co-occur in 2D views.
- A cross-attention mechanism, combining feature similarity with a normalized spatial prior, allows for end-to-end joint consideration of mask-induced features and spatial proximity for improved instance segmentation.
- Dynamic adjustment mechanisms for instance tokens, including pruning of useless/duplicate tokens and splitting of conflicted ones, are crucial for robustly managing instance variability and preventing over-segmentation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.