Reproduce “In the Eye of the Beholder: Gaze and Actions in First Person Video” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yin Li, Miao Liu, James M. Rehg

This research from the University of Wisconsin-Madison and Georgia Institute of Technology introduces a probabilistic deep learning model that jointly estimates gaze and recognizes actions from first-person videos. It also presents the EGTEA Gaze+ dataset, which includes synchronized video, gaze tracking, hand masks, and fine-grained action annotations. The model achieves improved action recognition accuracy and competitive gaze estimation performance, demonstrating robustness to gaze uncertainty and generalizability to datasets without explicit gaze supervision.

Key insights

  • Modeling gaze probabilistically significantly improves action recognition by robustly accounting for inherent uncertainty and noise in egocentric gaze measurements, outperforming deterministic gaze integration methods.
  • A comprehensive multi-modal dataset with synchronized gaze and action annotations provides crucial resources for advancing research in joint gaze estimation, action recognition, and hand segmentation in FPV.
  • The proposed attention mechanism can effectively learn to focus on action-relevant regions even without direct gaze supervision during training (using a uniform prior), demonstrating the model's adaptability and potential for broader application.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.