Reproduce “Trajectory World Models for Heterogeneous Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shaofeng Yin, Jialong Wu, Siqiao Huang, Xingjian Su, Xu He, Jianye Hao, Mingsheng Long
This paper presents TrajWorld, a novel Transformer-based world model architecture designed to handle the inherent heterogeneity in sensor and actuator information across diverse control environments. By pre-training on UniTraj, a large-scale, curated dataset, TrajWorld enables positive knowledge transfer and demonstrates improved performance in transition prediction, off-policy evaluation, and model predictive control on complex robotic tasks.
Key insights
- By decomposing state and action vectors into fundamental scalar "variates," heterogeneity across diverse control environments can be unified and processed by a single model.
- In-context learning within a Transformer enables the model to infer environment-specific dynamics and variate semantics directly from historical trajectories, eliminating the need for explicit environment IDs.
- Explicitly modeling both temporal dependencies within individual variates and relationships between different variates at a given timestep (via interleaved attention) is crucial for accurate and generalizable trajectory prediction.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.