Reproduce “UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Liming Tan, Ye Chen, Hao Zhang, Lirong Qian, Feifei Li, Bingbing Ni
UniMoCa uses a Motion-Camera Visual Proxy (MCVP) that combines identity-neutral rendered human motion and explicit camera markers as one visual control. It estimates trajectories, refines motion, renders the proxy, and conditions a Wan2.2 generator on its tokens. Against PRoPE, MCVP reports RotErr 1.13 versus 1.94 and TransErr 4.53 versus 5.14, without proving broad visual superiority.
Key insights
- Unifying disparate motion and camera controls into a single visual proxy (MCVP) allows the video generation model to interpret both types of information within a consistent spatio-temporal token space, reducing ambiguity.
- Explicitly rendering camera dynamics as visual markers within the proxy provides direct, interpretable cues for camera movement, facilitating more accurate control.
- Techniques like Shifted RoPE and MCVP Dropout are crucial for training robustness, enabling the model to effectively distinguish conditioning information from generated content and handle imperfect input signals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.