Reproduce “ReWorld: An Interactive World Model with Long-Horizon Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhifei Chen, Luozhou Wang, Guibao Shen, Dongyu Yan, Shuai Yang, Tianshuo Xu, Yihua Du, Wei Wang and 3 more
ReWorld develops interactive video generation that follows camera commands and supports revisits with a bounded cache. It trains control and memory with different attention windows, then retrieves earlier views by camera pose. In a 64-second test, its landmark cache achieves revisit SSIM of 0.3752 versus 0.3476 for sliding-window caching; this measures visual similarity at revisited poses, not general scene understanding.
Key insights
- Structurally separating control and memory learning through dedicated attention head configurations and random routing during training effectively resolves the tension between immediate action following and long-range spatial recall.
- A landmark-based memory consolidation system, combined with pose-indexed attention, allows for unbounded spatial memory rollouts while operating within fixed computational resources.
- Training with partial and non-contiguous historical contexts via chunk-drop training is crucial for ensuring robust model performance when memory is managed by selective retrieval at inference time.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.