Reproduce “RAVEN: Rethinking Adversarial Video Generation with Efficient Tri-plane Networks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Partha Ghosh, Soubhik Sanyal, Cordelia Schmid, Bernhard Schölkopf
RAVEN, a GAN-based model, proposes a tri-plane hybrid representation and an explicit optical flow motion model for unconditional video generation. It synthesizes photorealistic 256x256 pixel videos up to 160 frames, achieving an FVD16 of 97 on the "Talking Faces all Motions" dataset and reducing computational FLOPs by over 50% compared to previous state-of-the-art GANs.
Key insights
- Adapting the tri-plane representation, originally for 3D object modeling, to video data efficiently addresses memory and computational challenges while effectively capturing long-range spatio-temporal dependencies.
- Explicitly modeling motion using an optical flow mechanism in the latent feature space significantly improves consistency in object dynamics (translations, rotations), preventing feature duplication across time.
- Leveraging a high-capacity CNN backbone (e.g., StyleGAN-T) within the tri-plane framework, combined with progressive training and selective super-resolution, is crucial for achieving high-fidelity and long-duration video generation efficiently.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.