Reproduce “Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Runjia Qian, Zile Wang, Jihai Zhang, Kai Zou, Wei Yu, Jiaxing Li, Zexiang Liu, Yaokun Li and 9 more
Matrix-Game 3.5 generates streaming, camera-controlled video worlds using patch memory: it lifts past image patches into 3D with estimated depth and camera data, then retrieves them in the current view while generating unsupported regions. Tiled PRoPE combines camera geometry with temporal position encoding. On the Hard-Trajectory split, its 5B base model at 720p on one GPU has 2.70° rotation error, versus 8.34° for the runner-up; this compares systems, not individual components.
Key insights
- Integrating geometry-aware patch memory with frustum queries and z-buffering allows precise recall and rendering of static scene elements from novel viewpoints.
- Explicitly disentangling static and dynamic content at the memory encoding stage enhances consistency for both persistent environments and evolving subjects.
- A two-stage progressive distillation process can effectively transform complex diffusion models into efficient, real-time interactive systems, preserving long-horizon controllability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.