Reproduce “PhyCo: Learning Controllable Physical Priors for Generative Motion” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sriram Narayanan, Ziyu Jiang, Srinivasa Narasimhan, Manmohan Chandraker

PhyCo endows video diffusion models with the ability to generate motion that is continuously controllable across physical properties like friction, restitution, and applied force. By combining a large-scale physically-grounded simulation dataset with VLM-guided reward optimization, the system significantly enhances physical realism and precise control, reducing directional error in force-conditioned control by over 60% compared to a baseline.

Key insights

  • Explicitly training generative models with dense, pixel-level physical property maps, derived from diverse simulated data, enables continuous and interpretable control over complex motion dynamics.
  • Vision-Language Model (VLM)-guided reward optimization effectively refines the physical consistency and control fidelity of generated videos by providing semantic feedback during training.
  • Learning physical priors from carefully designed synthetic environments can generalize effectively to novel object categories, motion types, and appearance shifts in real-world videos.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.