Reproduce “EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT” on OpenResearch

About this paper

Baoqi Pei, Yifei Huang, Jilan Xu, Yuping He, Guo Chen, Fei Wu, Yu Qiao, Jiangmiao Pang

EgoThinker enhances Multimodal Large Language Models (MLLMs) with robust egocentric reasoning, leveraging a novel 5-million-sample dataset, EgoRe-5M, and a two-stage supervised and reinforcement fine-tuning curriculum. The framework achieves state-of-the-art performance across various egocentric benchmarks, including an 8.4% improvement on cross-view skill transfer, and significantly boosts fine-grained spatio-temporal grounding while maintaining general video understanding capabilities.

Key insights

• MLLMs can be effectively endowed with robust egocentric reasoning and precise spatio-temporal grounding capabilities through a carefully designed data collection and two-stage fine-tuning approach.
• The combination of supervised instruction tuning and reinforcement fine-tuning, particularly with rule-based reward functions for grounding, is crucial for simultaneously improving high-level reasoning and low-level localization in egocentric contexts.
• Explicit Chain-of-Thought rationales and fine-grained grounding annotations in large-scale egocentric datasets significantly improve a model's ability to understand complex human-object interactions and temporal causality.

Summary by alphaXiv.