Reproduce “Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shir Gur, Sagie Benaim, Lior Wolf
Researchers at Tel Aviv University developed Hierarchical Patch VAE-GAN (hp-VAE-GAN), the first method to unconditionally generate diverse and high-quality videos from just a single input video sample. This hybrid model leverages a patch-based VAE for structural diversity at coarse scales and a GAN for fine textural details at finer scales, effectively overcoming mode collapse.
Key insights
- A hybrid VAE-GAN architecture is effective for single-sample generation, with VAEs handling coarse structural diversity and GANs refining fine-grained details.
- Modeling internal spatio-temporal patch distributions from a single video, rather than whole video distributions, is crucial for preventing memorization and generating diverse content.
- Sequential training across hierarchical scales with appropriate network freezing prevents mode collapse and ensures stable, high-quality output.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.