Reproduce “SANeRF-HQ: Segment Anything for NeRF in High Quality” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yichen Liu, Benran Hu, Chi-Keung Tang, Yu-Wing Tai
SANeRF-HQ presents a framework for high-quality, prompt-based 3D object segmentation within Neural Radiance Fields, effectively integrating SAM's capabilities to produce accurate and multi-view consistent object masks. The method consistently outperforms prior approaches on diverse NeRF datasets in terms of mIoU and Accuracy by aggregating 2D SAM masks into a 3D object field and refining boundaries with a novel Ray-Pair RGB loss.
Key insights
- Aggregating 2D segmentation masks from multiple views into a unified 3D object field is crucial for achieving high multi-view consistency and accurate 3D segmentation in NeRFs.
- Refining segmentation boundaries by leveraging color similarity and focusing loss application on challenging regions (Ray-Pair RGB loss) can significantly enhance mask quality, even with continuous neural representations.
- Leveraging pre-computed high-quality 2D SAM features (feature caching) generally yields more accurate results than distilling them into a 3D feature field, though distillation offers efficiency for novel view rendering.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.