Reproduce “RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfer” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liu Liu, Xiaofeng Wang, Guosheng Zhao, Keyu Li, Wenkang Qin, Jiagang Zhu, Jiaxiong Qiu, Zheng Zhu and 2 more
RoboTransfer generates synchronized multi-view video that preserves demonstration geometry while changing object and background appearance. It conditions video generation on depth and surface normals, while reference images guide scene and object appearance. With a 50/50 real/synthetic data mix, spoon pick-and-place success under Diff-All rises from 13.3% to 46.7%, but performance declines at higher synthetic shares.
Key insights
- Jointly reasoning across multiple viewpoints during video generation is crucial for maintaining multi-view geometric consistency without modifying the diffusion backbone architecture.
- Explicitly conditioning diffusion models with precise geometric cues (metric depth and surface normals) significantly improves the geometric consistency and overall quality of synthetic data.
- Disentangled and fine-grained control over both background (via inpainting) and individual object appearances (via object-wise CLIP encoding) is critical for creating diverse, task-relevant synthetic training data that enhances policy generalization.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.