Reproduce “SpatialFusion: Endowing Unified Image Generation with Intrinsic 3D Geometric Awareness” on OpenResearch

About this paper

Haiyi Qiu, Kaihang Pan, Jiacheng Li, Juncheng Li, Siliang Tang, Yueting Zhuang

SpatialFusion introduces a framework that integrates a Spatial Transformer with Multimodal Large Language Models and Diffusion Transformers to imbue unified image generation models with intrinsic 3D geometric awareness. This approach achieved over 70% accuracy in object and camera pose tasks on the GenSpace benchmark and showed substantial improvements in spatial relation reasoning compared to prior methods.

Key insights

• While MLLMs excel at semantic understanding, their internal representations weakly encode 3D geometry; explicit integration of a specialized Spatial Transformer is necessary to derive robust 3D structural scaffolds.
• Incorporating derived 3D geometric cues directly into the diffusion model's latent space, before patchification, effectively guides the denoising process to achieve spatially coherent and accurate image synthesis.
• A progressive two-stage training strategy, including geometric-aware pre-training and geometry-guided joint training, enables the model to effectively learn and leverage intrinsic 3D geometric awareness without sacrificing generative quality.

Summary by alphaXiv.