Reproduce “DiT360: High-Fidelity Panoramic Image Generation via Hybrid Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoran Feng, Dizhe Zhang, Xiangtai Li, Bo Du, Lu Qi
DiT360, developed by Insta360 Research and universities, produces high-fidelity panoramic images by employing a hybrid training strategy that combines limited panoramic data with abundant perspective images and introduces multi-level geometry-aware supervision in the Diffusion Transformer framework. The method achieves state-of-the-art results across various metrics for realism, diversity, and perceptual quality while naturally supporting tasks like inpainting and outpainting.
Key insights
- Combining domain-specific panoramic data with general perspective data via a multi-level hybrid training approach effectively addresses data scarcity and improves overall generation quality.
- Explicitly integrating geometry-aware supervision, such as circular padding for boundaries and specialized losses for rotation and distortion in the latent space, is crucial for producing geometrically accurate and consistent panoramas.
- Refining existing panoramic datasets by inpainting polar regions with a perspective model can significantly improve the quality of training data and subsequent model performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.