Reproduce “MIND: Benchmarking Memory Consistency and Action Control in World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yixuan Ye, Xuanyu Lu, Yuxin Jiang, Yuchao Gu, Rui Zhao, Qiwei Liang, Jiachun Pan, Fengda Zhang and 2 more
Researchers developed MIND, a new open-domain closed-loop benchmark designed to evaluate the memory consistency and action control capabilities of world models across diverse visual environments and both first-person and third-person perspectives. This benchmark leverages high-fidelity Unreal Engine 5 data to assess how well models retain environmental context and execute precise actions.
Key insights
- Models utilizing context memory show improved short-term consistency, but current approaches struggle with maintaining truly long-term memory and reproducing coherent scene content during revisits.
- Precise action control remains a challenge for world models, with action accuracy deteriorating even with context memory and in-domain actions, and generalization often failing with altered action spaces.
- Third-person perspective modeling presents severe limitations for current world models, which struggle with accurate character-background interaction and maintaining consistent visual relationships.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.