Reproduce “Generative Visual Code Mobile World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Woosung Koh, Sungjun Han, Segyu Lee, Se-Young Yun, Jamin Shin
gWorld represents action-conditioned mobile GUI transitions as executable HTML/CSS rather than text descriptions or directly generated screenshots. Its 8B and 32B models achieve 74.9% and 79.6% average instruction accuracy on MWMBENCH, respectively, while keeping render-failure rates at 1.4% and 0.6%; the stronger world-model predictions also improve single-step policy selection.
Key insights
- Executable web code provides a middle ground between semantic text and unconstrained pixels: text, controls, spatial relationships, and styling are represented explicitly while remaining visually renderable.
- Perceptual image similarity can reward copying a nearly unchanged screenshot even when the action is ignored; gWorld’s lower similarity-to-correctness correlation indicates more action-responsive state transformation.
- Trajectory-based relabeling and reasoning traces generated with access to the target state outperform direct next-state prompting, and performance improves steadily as training data increases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.