Reproduce “DeforM: Reasoning-Guided Physics-Aware Video Generation via Spatial-Temporal Masking” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yunyi Li, Yu Qiao, Yaohui Wang, Xinyuan Chen

The DeforM framework introduces a reasoning-guided approach for image-to-video generation, synthesizing physically plausible deformation dynamics by explicitly localizing and steering attention to physics-critical regions. This framework, specifically DeforM-Injection, achieved an overall Quality Score of 73.17 and a Physical Score of 4.40, outperforming state-of-the-art baselines in generating complex deformations like melting and squeezing.

Key insights

  • Explicitly localizing and directing generative model attention to physics-critical regions is fundamental for synthesizing accurate and consistent deformation dynamics.
  • VLM-guided reasoning effectively bridges the semantic-spatial gap, enabling models to associate abstract physical instructions from text prompts with concrete pixel locations for target objects.
  • The synergistic modulation of both self-attention (for localized spatial-temporal consistency) and cross-attention (for semantic anchoring of physical knowledge) significantly refines the generation of complex deformations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.