Reproduce “Marionette: Predicting World States, Rendering Geometry, Painting Appearance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zian Meng, Zhen Li, Chuanhao Li, Qiang Li, Kaipeng Zhang
Marionette, a new framework for interactive game world modeling, disentangles dynamics from appearance by predicting an explicit 276-dimensional world state and using a zero-parameter deterministic renderer before synthesizing photorealistic observations. This approach demonstrates enhanced consistency and controllability, allowing long-horizon state-level failures to be directly repaired through rules without modifying the appearance generation model.
Key insights
- Decoupling world state dynamics from visual appearance generation through an explicit state and deterministic rendering guarantees structural consistency and metric exactness by construction, preventing error accumulation.
- The explicit, interpretable world state allows for direct control via action tokens and facilitates targeted repair of long-horizon failures by imposing rules directly on the state, such as terrain collision or entity separation.
- Routing visual appearance through the predicted explicit state and deterministic renderer maintains high visual fidelity, achieving FVD scores comparable to models driven by ground-truth pose while offering significant architectural advantages in consistency and controllability.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.