Reproduce “VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xinyu Liu, Ailing Zeng, Wei Xue, Harry Yang, Wenhan Luo, Qifeng Liu, Yike Guo
VFX Creator introduces a framework for generating animated visual effects from text prompts and static images, leveraging a Video Diffusion Transformer with spatial and temporal control. It significantly improves video quality, realism, and motion accuracy, achieving an FID-VID of 29.92 and an FVD of 752.95, while offering precise control over effect timing and instance-level manipulation.
Key insights
- Combining a specialized dataset (Open-VFX) with fine-grained spatial and temporal control mechanisms significantly enhances the generative capabilities of Video Diffusion Transformers for complex VFX.
- Integrating temporal signals directly into the text embedding space via 'timestamp tokens' and utilizing a learnable ControlNet for spatial conditioning through LoRA adapters enables precise instance-level and rhythm-controlled effects.
- The framework's data efficiency, achieving high-quality generation with minimal training videos, demonstrates the potential for robust learning of abstract visual effects without extensive data requirements.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.