Reproduce “MMOne: Representing Multiple Modalities in One Scene” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhifeng Gu, Bing Wang

The Spatial Intelligence Group at The Hong Kong Polytechnic University introduced MMOne, a framework that integrates multiple sensory modalities within a single 3D scene representation by disentangling shared and modality-specific information. This approach improves rendering quality for individual modalities, achieving average PSNR gains of 0.5 dB for RGB and 0.4 dB for thermal in RGB-Thermal tasks, and enhances open-vocabulary segmentation while significantly reducing the number of Gaussians required.

Key insights

  • Explicitly disentangling shared and modality-specific information is crucial for effectively managing inherent 'modality conflicts' arising from property and granularity disparities in multimodal scene representation.
  • Introducing modality-specific opacity and conditional pruning within a 3D Gaussian Splatting framework allows a single set of Gaussians to efficiently adapt to the unique physical properties and required granularities of diverse modalities.
  • Decomposing multi-modal Gaussians based on conflicting gradients prevents performance degradation across modalities and enables robust, scalable integration of an increasing number of distinct data types.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.