Reproduce “NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, Nan Duan
NÜWA, developed by researchers from Microsoft Research Asia and Peking University, introduces a unified multimodal pre-trained model for diverse visual synthesis tasks, encompassing text-to-image, text-to-video, and video prediction. The model leverages a 3D transformer with a novel 3D Nearby Attention mechanism, generating temporally coherent videos and realistic images, outperforming baselines like CogView on MSCOCO and improving FVD scores on Kinetics and BAIR datasets.
Key insights
- A unified 3D transformer architecture can effectively handle diverse visual synthesis tasks (image, video, text, sketch) within a single model.
- The 3D Nearby Attention (3DNA) mechanism significantly reduces computational complexity from O(n²) to O(n·k) for high-dimensional visual data while preserving local spatial-temporal coherence.
- Multi-task pre-training across text-to-image, video prediction, and text-to-video generation, combined with VQ-GAN for tokenization, enables robust and high-quality visual content creation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.