Reproduce “GazeNLQ @ Ego4D Natural Language Queries Challenge 2025” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Wei-Cheng Lin, Chih-Ming Lien, Chen Lo, Chia-Hung Yeh

Researchers from National Taiwan Normal University and National Sun Yat-sen University developed GazeNLQ, a framework that integrates estimated gaze information as a third modality into egocentric video Natural Language Queries. The framework achieves competitive performance on the Ego4D NLQ dataset, demonstrating the utility of gaze for enhancing temporal localization in first-person videos.

Key insights

  • Freezing the weights of the gaze estimator during the grounding model's fine-tuning phase yields better performance than joint training, suggesting the estimator generalizes better when stable.
  • Incorporating estimated gaze information as a third modality (alongside video and text) enhances temporal localization accuracy in egocentric Natural Language Queries (NLQ).
  • The concept of 'negative gaze embeddings' potentially offers additional valuable information by indicating where the wearer's attention is not directed.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.