Reproduce “H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dingyi Rong, Yue Shi, Chaofan Ma, Jiezhang Cao, Zongrui Wang, Zeyu Zhang, Yao Mu, Guangtao Zhai and 1 more
H2R-Bench provides a framework for systematically evaluating video world models' generation of robot manipulation videos from human demonstrations, developed by Shanghai Jiao Tong University and Shanghai Artificial Intelligence Laboratory. It reveals a weak association (Spearman "rho" = 0.14) between models' visual fidelity and their ability to accurately transfer human functional interactions and robot embodiment, indicating a challenge in cross-embodiment task translation.
Key insights
- A significant disparity exists between the general visual quality of generated videos and the models' actual capability for embodied human-to-robot manipulation transfer.
- Functional contact transfer and embodiment correctness are identified as primary bottlenecks for current video world models when performing cross-embodiment tasks.
- Generating for a dexterous hand generally leads to slightly better H2R transfer performance compared to a parallel-jaw gripper, potentially due to the dexterous hand's closer morphological resemblance to a human hand.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.