Reproduce “Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Francesco Ragusa, Michele Mazzamuto, Rosario Forte, Irene D'Ambra, James Fort, Jakob Engel, Antonino Furnari, Giovanni Maria Farinella
Ego-EXTRA is a 50-hour egocentric video-language dataset featuring real-time, bidirectional expert-trainee conversations during procedural tasks, created to advance intelligent wearable AI assistants. The associated human-validated VQA benchmark of over 15,000 questions reveals that state-of-the-art MLLMs achieve only approximately 33% accuracy, significantly below human performance of nearly 90%.
Key insights
- Natural expert-trainee dialogues are heavily grounded in visual context, utilizing pronouns and ambiguous references, which require deep visual and temporal understanding from an AI.
- Current state-of-the-art MLLMs demonstrate significant limitations in providing expert-level assistance, achieving accuracies far below human experts on realistic egocentric VQA tasks.
- Optimal video context for current MLLMs involves short, uniformly sampled clips, as longer video durations or single-frame inputs tend to degrade performance, suggesting challenges in temporal reasoning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.