Reproduce “What They Saw, Not Just Where They Looked: Semantic Scanpath Similarity via VLMs and NLP metric” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mohamed Amine Kerkouri, Marouane Tliba, Bin Wang, Aladine Chetouani, Ulas Bagci, Alessandro Bruno
This research introduces a framework that transforms eye-gaze scanpaths into natural language descriptions using Vision-Language Models (VLMs), enabling the quantification of semantic similarity between gaze patterns with NLP metrics. The work demonstrates that this semantic approach provides a complementary dimension to traditional spatial and temporal scanpath analysis, offering new insights into the content of visual attention.
Key insights
- Semantic similarity offers a distinct, non-redundant dimension for comparing scanpaths, capturing content-level agreements that are not reflected in purely geometric alignments.
- The method of providing visual context to the VLM (e.g., patch size vs. marker-based full image) significantly influences the quality and independence of semantic descriptions.
- Using appropriately sized image patches (e.g., 256px) for fixation encoding is crucial for generating stable and interpretable semantic descriptions, while full-image context can introduce 'semantic leakage'.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.