Reproduce “Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images” on OpenResearch

About this paper

JiaKui Hu, Shanshan Zhao, Qing-Guo Chen, Xuerui Qiu, Jialun Liu, Zhao Xu, Weihua Luo, Kaifu Zhang and 1 more

Omni-View trains a multiview scene-understanding model alongside generation of views and geometry. It uses texture and geometry modules, autoregressive view prediction, and a dense-to-sparse reference schedule; training freezes understanding while refining RGB, depth, and pose generation. Omni-View-7B scores 55.4 average on VSI-Bench versus 46.3 for BAGEL-7B-FT, but bundled changes do not isolate generation as cause; grounding and geometry accuracy remain limitations.

Key insights

• Explicit generative tasks, such as texture synthesis and geometry estimation, are shown to be effective in fostering a deeper 3D scene understanding by requiring the model to acquire fundamental geometric and spatiotemporal capabilities.
• A modular generative architecture, separating texture and geometry components, yields superior understanding performance compared to a monolithic design, highlighting the benefits of specialized generative modules.
• Techniques like autoregressive generation for spatiotemporal modeling and a Dense-to-Sparse curriculum for progressive input complexity are crucial for improving the robustness and accuracy of 3D understanding from multiview images.

Summary by alphaXiv.