Reproduce “Open-Vocabulary Gaze Object Prediction: Benchmark and Method” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Binglu Wang, Sensen Niu, Ying Chen, Guangyu Guo
Open-vocabulary gaze object prediction is formulated through DiSG, a real-image benchmark with 13,041 images, 17,447 gaze-object annotations, 86 categories, and over 30 human-centric scenarios. A decoupled system combines text-guided object discovery with a gaze-derived saliency map, while Gradient-Informed Selection Tuning improves adaptation: the full model reaches 32.8 mSoC overall and 29.5 on novel categories, with 40.3 novel-category AP50.
Key insights
- Semantic recognition and gaze localization need not be solved by a single prediction head: language grounding can enumerate possible targets, while gaze saliency can choose among them.
- Selective parameter updates provide a better adaptation trade-off than updating the entire detector; a 40% selection ratio outperforms full tuning on novel-category GOP and detection.
- Fine-grained attention labels and targets outside object boxes expose challenges beyond standard object detection, including ambiguous gaze, occlusion, tiny objects, and body-part boundaries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.