Reproduce “SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ziyi Wu, Nikita Dvornik, Klaus Greff, Thomas Kipf, Animesh Garg
SlotFormer introduces an unsupervised method for visual dynamics simulation, combining object-centric representations with an autoregressive Transformer. This model achieves superior long-term video prediction quality, maintains consistent object dynamics, and improves performance on downstream tasks like visual question answering and action planning without requiring explicit object labels.
Key insights
- A unified Transformer architecture can effectively model both spatial object interactions and their temporal evolution simultaneously by processing concatenated object slots from multiple frames.
- Training with an autoregressive rollout, where the model's own predictions are used as input, is crucial for mitigating train-test discrepancies and achieving robust, high-quality long-term dynamics generation.
- Unsupervisedly learned object-centric dynamics models can transfer foundational knowledge to significantly boost performance on supervised downstream tasks (VQA, planning) without requiring task-specific object-level annotations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.