Reproduce “MemoBench: Benchmarking World Modeling in Dynamically Changing Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Haoyu Chen, Kaichen Zhou, Hang Hua, Kaile Zhang, Jingwen Qian, Wufei Ma, Haonan Chen, Chunjiang Liu and 6 more
A new diagnostic benchmark, MemoBench, evaluates the world modeling capabilities of generative models, focusing on memory consistency in dynamically changing, partially observable environments. It revealed that current state-of-the-art models struggle significantly with object permanence, achieving Object Reappearance Scores no higher than 0.6 and exhibiting limited semantic understanding of object state after occlusion.
Key insights
- Explicit camera conditioning alone does not guarantee object permanence; models require dedicated training to maintain consistent representations of objects that move out of the field of view.
- Standard evaluation metrics can be misleading, as model behaviors like camera inactivity can artificially inflate scores for visual quality and consistency.
- Current models demonstrate a trade-off between geometric fidelity and perceptual quality, indicating that achieving both accurate scene structure and visually natural frames remains an optimization challenge.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.