Reproduce “Efficient Encoder-Free Fourier-based 3D Large Multimodal Model” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Guofeng Mei, Wei Lin, Luigi Riz, Yujiao Wu, Yiming Wang, Fabio Poiesi

Fase3D is a scene-level 3D large multimodal model that avoids a dedicated pretrained 3D vision encoder. It combines superpoint grouping, multi-curve spatial serialization, localized FFT mixing, sparse graph pooling, and Fourier-augmented LoRA to reduce visual-tokenization cost while retaining competitive 3D reasoning performance; on ScanQA, it reaches up to 91.74 CIDEr with Vicuna-7B and uses 10.54 million activated visual parameters versus 58.26 million for 3D-LLaVA.

Key insights

  • Superpoint pooling and spectral context mixing complement each other: in ScanQA ablations, CIDEr rises from 76.04 with downsampled raw points to 79.70 with superpoints and 86.91 when FFT enhancement is added.
  • Four space-filling-curve traversals provide most of the benefit; additional traversals offer limited gains, suggesting a practical trade-off between spatial coverage and processing cost.
  • Fourier adaptation is most effective when applied to the visual pathway: adding the residual filter to both visual and text branches performs worse than visual-only filtering.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.