Reproduce “Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gengshan Yang, Andrea Bajcsy, Shunsuke Saito, Angjoo Kanazawa

The Agent-to-Sim (ATS) framework learns interactive 3D behavior models of agents from casual, longitudinal monocular RGBD videos. It reconstructs persistent 4D representations of agents and environments using multi-video registration and generates behaviors that react to observer motion and 3D scene context, achieving superior camera registration and end-to-end behavior prediction compared to baselines.

Key insights

  • Robust and persistent 4D spacetime representations of deformable agents and rigid environments can be built from casual, longitudinal monocular videos by leveraging large image models for multi-video registration and feature-metric refinement.
  • Decomposing interactive behavior generation into a hierarchical diffusion process (goal "," path "," body pose) significantly improves prediction accuracy and simplifies learning complex, multi-modal behavior distributions.
  • Conditioning behavior models on an egocentric perception of the scene and observer motion enables the generation of interactive, context-aware, and spatially controlled behaviors that generalize effectively to novel situations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.