Reproduce “IndEgo: A Dataset of Industrial Scenarios and Collaborative Work for Egocentric Assistants” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vivek Chavan, Yasmina Imgrund, Tung Dao, Sanwantri Bai, Bosong Wang, Ze Lu, Oliver Heimann, Jörg Krüger
The IndEgo dataset provides 197.1 hours of egocentric and 96.8 hours of exocentric multimodal data from authentic industrial settings, featuring collaborative work and long-horizon tasks, collected using Meta Project Aria devices. Baseline evaluations reveal that state-of-the-art multimodal models achieve limited performance on benchmarks like mistake detection (23.3-40.9% F1) and reasoning-based QA (57.6-64.1% accuracy), highlighting the complexity of industrial egocentric AI challenges.
Key insights
- Current state-of-the-art multimodal language models (VLMs) demonstrate limited performance on industrial egocentric tasks, indicating the significant challenges of these environments.
- Exocentric camera views and multimodal cues like audio and eye gaze provide valuable, context-dependent information, with audio proving essential for understanding collaborative tasks.
- The dataset's complexity, particularly in differentiating egocentric and co-worker actions in collaborative settings, reveals a major hurdle for current AI in analyzing human-human dynamics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.