Reproduce “Beyond Instance Slots: Semantically Rich World Models for Physical Interaction Planning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Juntao Cheng, Jingkai Wang, Yijun Shen, Xiansheng Chen, Zhiwei Yu

A Semantically Rich World Model (SR-WM) is introduced for robotic manipulation, explicitly binding visual entities to task-conditioned functional roles to guide action generation and selection. This results in an 83.0% closed-loop success rate across diverse LIBERO simulation tasks and improved cross-task generalization.

Key insights

  • Explicitly structuring world states into task-conditioned functional roles significantly enhances the quality of generated actions and the reliability of planning decisions in manipulation tasks.
  • Direct supervision of granular semantic transitions, such as contact, grasp, and predicate establishment/preservation, enables the model to learn causal temporal ordering and distinguish between effective and ineffective actions.
  • Integrating semantic understanding across action generation, selection, and repair mechanisms yields robust closed-loop control and facilitates strong generalization capabilities across a variety of manipulation tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.