Reproduce “From Virtual Games to Real-World Play” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wenqiang Sun, Fangyun Wei, Jinjing Zhao, Xi Chen, Zilong Chen, Hongyang Zhang, Jun Zhang, Yan Lu

RealPlay introduces a neural network-based real-world game engine capable of generating photorealistic, interactive video from user control signals. This system effectively transfers control learned from virtual game data to diverse real-world entities, such as vehicles, bicycles, and pedestrians, without requiring explicit real-world action labels, achieving a 90.0% control success rate.

Key insights

  • Control signals learned from high-fidelity game data can be effectively transferred to generate controllable real-world video, even for different entities like bicycles and pedestrians, without requiring explicit real-world action labels.
  • The use of an adaptive LayerNorm mechanism for action fusion proved more effective than text prompts in transferring control from game to real-world scenarios, achieving a higher control success rate.
  • Entities with larger motion amplitudes in real-world datasets tend to be easier for the model to control, and including diverse unlabeled real-world entity data during training improves generalization and control across various real-world entities.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.