Reproduce “EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yuan-Ming Li, Wei-Jin Huang, An-Lan Wang, Ling-An Zeng, Jing-Ke Meng, Wei-Shi Zheng
Researchers at Sun Yat-sen University introduce EgoExo-Fitness, a new dataset featuring synchronized egocentric and exocentric videos of full-body fitness actions with comprehensive annotations for temporal localization and interpretable quality assessment. This resource enables the development of AI models for tasks like guidance-based execution verification, crucial for AI-assisted fitness coaching.
Key insights
- A substantial domain gap exists between egocentric and exocentric full-body action views, causing significant performance drops when models trained on one view are applied to the other.
- Egocentric full-body action understanding is more challenging than exocentric, attributed to lower body visibility and ambiguous visual patterns in first-person perspectives.
- Simply combining egocentric and exocentric data during training does not consistently improve performance across both views, indicating a need for advanced cross-view learning strategies.
- An InfoNCE-based alignment loss in the GEVFormer model can effectively bridge the cross-view domain gap for guidance-based execution verification, particularly enhancing egocentric performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.