Reproduce “$Z$-sight: Unifying Vision-Language-Action and Latent World Modeling” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Junjie Ye, Yiqi Zhao, Muchen Xu, Celina Shiyu Wang, Hao Dong, Ruohai Ge, Rong Xue, Benjamin J Kolligs and 7 more

Z-sight predicts future spatial visual tokens in its vision-language model’s own representation and uses them to guide robot actions. An EMA teacher encodes later observations for training; foresight and action experts jointly refine predictions and actions at inference without generating video. On six Unitree G1 tasks it succeeded in 44/60 trials (73.3%), versus Fast-WAM’s 53.3%; this does not establish action-conditioned planning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.