Reproduce “Clockwork Variational Autoencoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vaibhav Saxena, Jimmy Ba, Danijar Hafner
The Clockwork Variational Autoencoder (CW-VAE) presents a hierarchical latent dynamics model with fixed, multi-speed temporal abstraction for long-horizon video prediction. It outperforms existing state-of-the-art models in open-loop predictions across various datasets and shows that slower hierarchical levels represent global, slow-changing information, while faster levels capture local, rapid details.
Key insights
- A simple fixed-speed temporal abstraction mechanism within a hierarchical latent model is highly effective for extending the duration of accurate video predictions.
- Slower-ticking, higher hierarchical levels learn to represent global, slowly changing information (e.g., scene textures, object identities), while faster, lower levels capture local, rapidly changing details (e.g., camera pose, object positions).
- The CW-VAE can dynamically adapt its allocation of information across hierarchical levels to align with the inherent temporal frequencies and speeds present in the input data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.