Reproduce “Controlling Space and Time with Diffusion Models” on OpenResearch

About this paper

Daniel Watson, Saurabh Saxena, Lala Li, Andrea Tagliasacchi, David J. Fleet

4DiM, a cascaded diffusion model from Google DeepMind and academic collaborators, synthesizes realistic, temporally consistent 4D scenes with simultaneous control over camera pose and time from one or more input images. The model demonstrates state-of-the-art performance in 3D novel view synthesis, achieving significantly lower FID and FDD scores compared to baselines, and effectively generates dynamic content through multi-frame conditioning.

Key insights

• Calibrating SfM-based datasets to metric scale significantly improves a generative model's ability to learn spatial regularities and achieve precise camera control, leading to better in-domain and out-of-distribution performance.
• Co-training with large-scale unposed video data is crucial for regularizing the model's understanding of spatiotemporal dynamics, enabling it to infer plausible geometry for unknown objects and improve generalization.
• Multi-guidance, which applies different guidance weights to individual conditioning variables, provides finer control over generative outputs, effectively balancing sample quality, pose alignment, and temporal dynamics.

Summary by alphaXiv.