Reproduce “Can World and Action Modeling Benefit Each Other under Unified Causal Training?” on OpenResearch

About this paper

Yue Su, Bokai Lin, Xinkai Wang, Chubin Zhang, Xiangnan Wu, Kuan Zhang, Weiyang Jin, Jiwen Yu and 4 more

OneWM tests whether joint video and action learning improves robot policies and action-conditioned prediction. It trains a temporally causal model with clean specified actions, partially clean actions, and noisy actions. On LIBERO-Plus it reports 59.27% success versus 52.50% for WAM_noisy; on LIBERO, PSNR is 22.95 dB versus 22.33 dB. These results support reciprocal benefits in tested settings, not broader superiority.

Key insights

• Explicitly predicting visual consequences from clean, specified actions can provide useful dynamics supervision for action generation; action learning can also improve shared representations used for prediction.
• The balance between prediction and action-generation training matters: a mixture with substantially more clean-action data reduced LIBERO-Plus success from 59.27% to 38.01%.
• Chaining generated actions and predicted frames for 16 imagined rounds demonstrates composability, but does not establish reliable long-horizon closed-loop control.

Summary by alphaXiv.