Reproduce “Taming Stable Diffusion for Text to 360° Panorama Image Generation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Cheng Zhang, Qianyi Wu, Camilo Cruz Gambardella, Xiaoshui Huang, Dinh Phung, Wanli Ouyang, Jianfei Cai

PanFusion, a dual-branch diffusion model from Monash University, Shanghai AI Laboratory, and Building 4.0 CRC, generates 360-degree panoramic images from text by integrating global panoramic context with local perspective view generation, addressing issues like loop inconsistency and repetitive elements. The model demonstrates superior consistency and quality, achieving a FAED of 6.04 and improving 3D IoU to 68.46 for layout conditioning.

Key insights

  • Combining global contextual reasoning (panorama branch) with detailed local generation (perspective branch) is effective for high-quality panoramic image synthesis.
  • Geometrically aware attention mechanisms, such as Equirectangular-Perspective Projection Attention (EPPA) with spherical positional encoding and projection masks, are crucial for maintaining consistency across different view projections.
  • Leveraging and adapting pre-trained large image models (like Stable Diffusion) with lightweight additions (LoRA, EPPA) can efficiently extend their capabilities to specialized geometric domains like panoramas, even with limited domain-specific data.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.