Reproduce “EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real World” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Heqian Qiu, Zhaofeng Shi, Lanxiao Wang, Huiyu Xiong, Xiang Li, Hongliang Li
The EgoMe dataset provides a novel resource for robot imitation learning by capturing the human observation-to-imitation process from an imitator's dual perspective. It features 15,804 paired videos across 41 real-world scenarios with 184 activity categories, alongside synchronized eye gaze, IMU data, and extensive multi-modal annotations, enabling new benchmarks for advanced embodied AI.
Key insights
- Directly modeling the imitator's 'observation-to-imitation' sequence from their perspective is crucial for developing AI agents that learn by observing, rather than just aligning multi-view data of a single actor.
- Synchronized multi-modal data, such as eye gaze and IMU, provides vital information about visual attention and body posture, enriching the understanding of the imitation learning process.
- A diverse array of real-world scenarios and activity categories is essential for training generalizable imitation learning models that can adapt to complex, unconstrained environments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.