Reproduce “SAP: Segment Any 4K Panorama” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Lutao Jiang, Zidong Cao, Weikai Chen, Xu Zheng, Yuanhuiyi Lyu, Zhenyang Li, Zeyu HU, Yingda Yin and 9 more
Segment Any 4K Panorama (SAP) adapts the Segment Anything Model 2 (SAM2) for promptable instance segmentation on high-resolution 360° panoramas by reformulating the input as a fixed-trajectory perspective video. This approach achieved an average of +17.2 mIoU gain over vanilla SAM2 on real-world 4K panoramic images.
Key insights
- Aligning the structural properties of panoramic inputs with the inductive biases of streaming memory models, by creating a pseudo-video from a fixed scanning trajectory, effectively addresses performance degradation.
- Inter-frame overlap (e.g., 50%) during the perspective scanning process is crucial for stable training and effective operation of SAM2's temporal memory bank, ensuring consistent instance tracking.
- Training with a large-scale synthetic 4K panoramic dataset, mixed with original SAM2 data, is vital for adapting the model to the specific distortions of panoramas while preserving its generalizability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.