Reproduce “Beyond Placement and Articulation: Usage-Driven Code Scenes for Embodied Interaction” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zijian Xiao, Zipeng Ye, Jinkun Hao, Xiong Yang, Yuchen Xie, Ran Yi

RoomWright generates interactive simulation scenes by reasoning from activities about needed objects and encoding executable interactions between them. It represents triggers, conditions, and effects so pressing a button can change a display’s state. RoomWright reports 16.4 interactive objects versus 4.4 for SceneCode, but this does not establish robot task success or a measured advantage on complex-interaction metrics.

Key insights

  • Usage-driven object reasoning, with task-centric expansion, is crucial for generating functionally rich, plausible, and contextually appropriate scene compositions for embodied agents.
  • Representing scenes as executable code enables the explicit encoding and simulation of complex, causally linked multi-part and cross-object interactions, extending beyond simple object articulation.
  • Leveraging part-level semantic annotations and Vision-Language Model (VLM) guidance provides an effective method for resolving ambiguity and accurately orienting manipulable objects based on their intended usage.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.