Reproduce “WorldSimProbe: Diagnosing Simulator Faithfulness in Action-Conditioned World Models for Embodied Manipulation” on OpenResearch

About this paper

Peterson Co, Sicheng Hu, Chunxuan Jiao, Hongyang Cheng, Yulin Luo, Yijie Xu, Sixiang Chen, Zhongxia Zhao and 12 more

WorldSimProbe evaluates action-conditioned world models as simulators, testing whether supplied controls produce matching robot motion and grounded environmental changes. Using five probes, it assesses six models on 18,608 instances across three platforms. Motion fidelity falls as control mismatch rises, while interaction scores vary by primitive. In one RoboTwin task, generated-data policies had lower success on out-of-distribution controls, a result limited to that case.

Key insights

• Simulator faithfulness for ACWMs requires precise action-to-motion correspondence and physically grounded environment responses, extending beyond mere visual plausibility or task success.
• Granular diagnostic suites can effectively localize specific failure modes in ACWMs, revealing weaknesses in action calibration, generalization, and interaction dynamics that aggregate metrics often obscure.
• ACWMs exhibit systematic biases, including attenuated responses to local action perturbations, reliance on motion priors for out-of-distribution trajectories, and a tendency for contact hallucination when visual cues are misleading.

Summary by alphaXiv.