Reproduce “Gaze-Regularized VLMs for Ego-Centric Behavior Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anupam Pani, Yanchao Yang

Researchers from The University of Hong Kong developed a gaze-regularized framework for Vision Language Models, integrating human eye gaze to significantly improve the accuracy of predicting future activities and providing fine-grained descriptions of current actions in ego-centric videos. This approach achieved nearly a 13% semantic score improvement for future activity prediction over baselines.

Key insights

  • Human eye gaze, when systematically integrated, serves as a powerful modality for inferring short-term human goals, leading to more fine-grained and actionable predictions from VLMs.
  • Aligning a VLM's internal attention with human gaze patterns through explicit regularization is more effective than merely providing gaze information as raw input or simple visual overlays.
  • The granularity and quality of textual annotations are crucial; more detailed descriptions significantly amplify the performance benefits derived from gaze integration in VLMs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.