Reproduce “ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qineng Wang, Wenlong Huang, Yu Zhou, Hang Yin, Tianwei Bao, Jianwen Lyu, Weiyu Liu, Ruohan Zhang and 3 more
ENACT introduces a benchmark for evaluating embodied cognition in Vision-Language Models through two VQA-based sequence reordering tasks: forward and inverse world modeling. The work reveals that current VLMs struggle with long-horizon interactions and exhibit human-like biases, performing significantly below human levels in predicting and inferring action-observation sequences in complex environments.
Key insights
- Current VLMs struggle significantly with long-horizon embodied world modeling, with performance degrading sharply as the number of interaction steps increases, indicating a lack of robust interactive memory and state tracking.
- VLMs demonstrate an asymmetry in reasoning, performing better on inverse world modeling (retrospective action inference) than on forward world modeling (prospective visual simulation).
- VLMs exhibit anthropocentric biases, performing worse when camera parameters (FOV, height) deviate from human-like vision, and showing a strong right-handed bias in understanding actions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.