Reproduce “Open-Sora: Democratizing Efficient Video Production for All” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li and 1 more
HPC-AI Tech's Open-Sora offers an open-source framework for efficient video production, reproducing techniques from advanced proprietary models to generate high-fidelity video content up to 16 seconds at 720p. The model achieved a VBench total score of 79.76%, establishing a new benchmark for open-source video generation capabilities.
Key insights
- A decoupled Spatial-Temporal Diffusion Transformer (STDiT) with enhancements like QK-normalization enables stable and high-performance video generation when scaling transformer architectures.
- An efficient 3D autoencoder, built upon a pre-trained 2D VAE, effectively compresses video data for temporal understanding without sacrificing original frames per second (FPS).
- Detailed data curation, including aesthetic and motion scoring, alongside comprehensive multi-modal captioning, is crucial for training robust and controllable generative video models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.