Reproduce “WorldMemBench: A Comprehensive Benchmark for Memory of Video World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Nan Zhou, Jianxiong Gao, Qiuyu Wang, Wen Wang, Jiahao Wang, Yufeng Yuan, Shuaiting Li, Hao Ouyang and 5 more
The authors introduce WorldMemBench, a benchmark for testing whether generated video worlds preserve scenes and continue events while users look away. It directs models through planned camera departures and returns, assessing static continuity against the initial view and dynamic event progression separately. Across 260 cases and nine models, static and dynamic performance diverged; results measure benchmark performance, not its causes.
Key insights
- Static retention and dynamic state evolution do not move together: HY-WorldPlay leads all seven static-memory metrics, while LingBot World and YUME score higher on overall dynamic memory.
- Repeated revisits reveal accumulating losses that a first return would not capture; changing the exploration axis can add further difficulty, though the effect varies across models and metrics.
- Context selection involves trade-offs across memory dimensions: view-relevant retrieval improves several human and dynamic measures over full history, but full history performs better on some object and environment measures.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.