Reproduce “Simple Video Generation using Neural ODEs” on OpenResearch

About this paper

David Kanaa, Vikram Voleti, Samira Ebrahimi Kahou, Christopher Pal

Researchers from Polytechnique Montréal and Mila introduced a video generation framework combining Neural Ordinary Differential Equations (Neural ODEs) with an encoder-decoder architecture to model continuous-time dynamics in latent space. This approach, particularly with a stochastic recurrent neural network encoder, successfully generates smooth video frames and extrapolates future sequences from training based on reconstruction on the Moving MNIST dataset.

Key insights

• Enforcing continuous latent dynamics through Neural ODEs intrinsically leads to smoother generated video motion, addressing a common challenge in video generation.
• Utilizing a stochastic RNN encoder, which processes multiple conditioning frames, substantially improves the model's performance and generalization, capturing both structural and temporal information more effectively.
• The model can successfully extrapolate to predict future frames even when primarily trained on a reconstruction objective (reconstructing initial frames), suggesting that Neural ODEs effectively capture underlying continuous dynamics.

Summary by alphaXiv.