Reproduce “Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haodong Yan, Hang Yu, Zhide Zhong, Weilin Yuan, Xin Gong, Zehang Luo, Chengxi Heyu, Junfeng Li and 3 more
SCAR generates hand-object interaction videos from a still image and instruction, jointly producing video and a compact interaction representation with hand/object contours, an estimated contact cue, and depth. On Taco’s egocentric test setting, SCAR_W scores 8.899 on VBench’s weighted total, above Wan2.1’s 8.792 under matched fine-tuning and checkpoint conditions. This supports higher video-quality and alignment scores, not direct physical accuracy.
Key insights
- A combined representation of contact-augmented hand-object contours and depth maps provides both fine-grained interaction cues and holistic 3D structural context in a scalable manner.
- Integrating HOI representation generation and video synthesis into a single, unified diffusion process effectively overcomes error accumulation issues inherent in multi-stage approaches.
- Learning shared interaction semantics across visual and representation modalities, followed by specialized refinement, is effective for guiding generative models towards physics-realistic interactions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.