Reproduce “EgoProceVQA: A Novel Egocentric Procedural Understanding Task with Self-Skill-Exploration Agent” on OpenResearch

About this paper

Junlong Li, Junxi Li, Yuxiang Yang, Wenbin Zou, Lap-Pui Chau, Yi Wang

Researchers introduced EgoProceVQA, a new benchmark for egocentric procedural understanding, which evaluates models across six key-step-centric video question answering dimensions. An accompanying training-free agent, EgoProceAgent, achieved state-of-the-art performance among open-source 7B/8B models on several tasks, improving its base model's accuracy on procedural sequence reasoning by 84.3%, although a substantial gap remains compared to human capabilities.

Key insights

• State-of-the-art MLLMs, both open and closed-source, exhibit substantial limitations in complex key-step-level procedural understanding, particularly in temporal grounding and step ordering tasks.
• Increasing input frames from 8 to 16 or applying standard Chain of Thought prompts did not yield consistent performance improvements for most models on procedural tasks.
• The proposed EgoProceAgent demonstrates that decoupling reasoning via autonomously discovered, task-specific skill strategies is more effective for zero-shot procedural understanding than relying on direct MLLM reasoning.

Summary by alphaXiv.