Reproduce “3UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding” on OpenResearch

About this paper

Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, Huchuan Lu

An end-to-end multimodal large language model, 3UR-LLM, directly processes 3D point clouds for scene understanding, outperforming prior state-of-the-art on ScanQA with a 7.1% CIDEr increase and reducing total training time by 49%. It also introduces 3DS-160K, a 160K sample 3D-text dataset generated via an automated pipeline.

Key insights

• High-quality 3D-text instruction data can be cost-effectively generated at scale using open-source 2D MLLMs and LLMs, mitigating data scarcity.
• Direct integration of 3D point cloud processing via a transformer-based object detector is more efficient and accurate than indirect 2D-to-3D reconstruction.
• A well-designed 3D compression module, coupled with query fusion, is critical for efficient processing of extensive 3D features and enhancing object-level spatial understanding.
• Fine-tuning the 3D compressor's weights, rather than freezing them, is essential for optimal performance.

Summary by alphaXiv.