Reproduce “Controllable Video Generation by Learning the Underlying Dynamical System with Neural ODE” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yucheng Xu, Li Nanbo, Arushi Goel, Zijian Guo, Zonghai Yao, Hamidreza Kasaei, Mohammadreze Kasaei, Zhibin Li
TiV-ODE introduces a framework for controllable video generation by integrating static image inputs with text captions and modeling continuous dynamical systems using Neural Ordinary Differential Equations. This approach allows for generating videos with flexible frame rates and specified motions, while achieving faster inference compared to sequential generation methods.
Key insights
- Fusing image and text inputs provides a comprehensive and high-level control mechanism over both the visual content and dynamic motions within generated videos.
- Modeling temporal evolution as a continuous dynamical system via Neural ODEs allows for generating videos with flexible frame rates and accurate slow-motion effects, moving beyond discrete frame-by-frame predictions.
- A non-auto-regressive generation approach, where all frames are derived by solving a continuous ODE from an initial condition, can significantly boost inference speed and computational efficiency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.