Reproduce “MV-Adapter: Multi-view Consistent Image Generation Made Easy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zehuan Huang, Yuan-Chen Guo, Haoran Wang, Ran Yi, Lizhuang Ma, Yan-Pei Cao, Lu Sheng
MV-Adapter generates compatible views of an object from text or a single image. It adds separately trained attention modules while retaining the image generator’s original attention and weights. On an SDXL image-to-multiview ablation, parallel attention outperformed serial attention on PSNR, SSIM, and LPIPS; this result does not establish equal performance across models, inputs, or viewpoints.
Key insights
- The adapter-based approach efficiently injects and learns 3D geometric consistency by only optimizing a small number of parameters, constraining the optimization space and preserving the base model's generative prowess.
- Decoupling new multi-view and image cross-attention layers by duplicating and zero-initializing existing spatial self-attention weights ensures that the adapter initially has no impact on the original feature space, allowing for gradual, non-disruptive learning.
- A parallel attention architecture is crucial for effectively inheriting and leveraging the powerful image priors from the pre-trained spatial self-attention layers, leading to more efficient learning of new geometric knowledge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.