Reproduce “SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guanting Ye, Qiyan Zhao, Wenhao Yu, Liangyu Yuan, Mingkai Li, Xiaofeng Zhang, Jianmin Ji, Yanyong Zhang and 2 more

SpatialSoPE adds geometric and directional position cues to point-cloud tokens in SpatialLM while retaining their sequence order. It combines sequence indices with spherical coordinates and multi-scale frequency mixing, changing positional encoding rather than the point-cloud encoder or language model. On ARKitScenes detection, it reaches 63.2 F1 at 3D IoU 0.50 versus 60.7 for SpatialLM; gains are benchmark- and model-specific.

Key insights

  • Explicitly modeling 3D geometry and directional information using spherical coordinates significantly improves cross-modal attention and overall spatial perception in transformer-based LVLMs.
  • Optimally allocating frequency bandwidths, particularly dedicating higher frequencies to angular components, is critical for effectively encoding fine-grained spatial and directional nuances.
  • Combining multi-scale positional encoding (linear, logarithmic, periodic) at the phase level effectively disentangles and represents both local and global 3D structures, leading to more balanced attention and robust performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.