Reproduce “ContactFlow: A video action conditioning that transfers across embodiments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sami Azirar, Enrico Pallotta, Jan Nogga, Jürgen Gall, Sven Behnke, Hermann Blum
ContactFlow conditions a video world model on object-side contact traces rather than the actor’s appearance or joint motion. It estimates sparse 3D contact points and local displacement, projects them into image space, and trains on human and robot interactions. On 25 held-out DROID clips, DreamSim was 0.039 versus 0.043 for Kinema4D, but other metrics favored Kinema4D.
Key insights
- Abstracting manipulation dynamics to object-centric 3D contact point trajectories creates a minimal and universal action representation that effectively transfers across human hands and various robot grippers.
- Conditioning a video generative model with Contact Flow enables the prediction of physically plausible manipulation outcomes, moving beyond mere visual coherence to respect underlying interaction mechanics.
- A world model capable of predicting physically realistic interactions can serve as a zero-shot verifier for proposed robot trajectories, significantly enhancing the robustness and safety of robot control in unseen environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.