Reproduce “ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yizhan Li, Jianxin You, Mengyang Xiong, Yinhuan Chen, Zicheng Zhao, Dekun Wu, Dongqing Zhang, Bang Liu
ReactHuman introduces a physics-grounded benchmark for testing whether multimodal language models can convert brief visual observations of sudden hazards into safe, executable reactions for a simulated humanoid. Across 2,138 model decisions, the evaluated systems averaged 54.0% semantic action accuracy and 80.8% safety validity, while showing large gaps in spatial execution, speed sensitivity, and adaptation to appearance–physics conflicts.
Key insights
- Choosing the right response and carrying it out effectively are distinct abilities: models averaged 54.0% action accuracy, but their average hand endpoint remained 1.23 meters from the true impact point and action–intent alignment was only 0.47.
- Most models displayed fixed behavioral preferences rather than balanced reasoning across situations: six generally favored retreating, while Claude Opus 4.8 favored catching, and no evaluated model performed strongly on both Catch and Dodge decisions.
- Models were weakly responsive to quantitative motion and physical evidence, often freezing during slow hazards and treating objects according to appearance even when their simulated material behavior contradicted that appearance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.