Reproduce “EgoExo-WM: Unlocking Exo Video for Ego World Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Danny Tran, Roberto Martín-Martín, Kristen Grauman
The EgoExo-WM framework demonstrates a method to enable egocentric world models to learn from abundant exocentric video by synthesizing action-aligned egocentric views. This approach reduces L2 embedding error by over 50% on certain datasets and improves goal-directed planning for 3D human motion compared to models trained solely on egocentric data.
Key insights
- Abundant exocentric video can be effectively utilized to scale egocentric world model training, provided the data is appropriately transformed into an action-aligned egocentric format.
- Explicitly incorporating 3D human kinematic priors during exocentric-to-egocentric video conversion is crucial for synthesizing high-quality, action-consistent egocentric data, overcoming limitations of prior view synthesis methods.
- Training a world model with this expanded, action-aligned data, leveraging 3D human motion, substantially improves both future visual prediction and goal-directed planning for embodied agents.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.