Reproduce “PanoWorld: Towards Spatial Supersensing in 360$^\circ$ Panorama World” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Changpeng Wang, Xin Lin, Junhan Liu, Yuheng Liu, Zhen Wang, Donglian Qi, Yunfeng Yan, Xi Chen
PanoWorld introduces a framework that enhances Multimodal Large Language Models (MLLMs) to directly reason about 360-degree panoramic environments by explicitly integrating spherical geometry into their visual processing. The framework achieved an overall accuracy of 56.5% on its PanoSpace-Bench diagnostic benchmark and demonstrated 70.0% performance on the H*Bench human-centric visual search task after fine-tuning.
Key insights
- Explicitly embedding spherical geometry into the MLLM's visual processing via a cross-attention mechanism is crucial for robust pano-native spatial reasoning.
- A comprehensive metadata construction pipeline, featuring both geometric detection verification and semantic verification, is essential for generating high-quality and verifiable panoramic instruction-tuning data.
- Dedicated pano-native spatial learning, including fine-tuning across all MLLM components (vision encoder, vision-language interface, and language model), provides a transferable spatial initialization for downstream tasks that is superior to prompting or task-specific fine-tuning alone.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.