Reproduce “From Gaze to Guidance: Interpreting and Adapting to Users' Cognitive Needs with Multimodal Gaze-Aware AI Assistants” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Valdemar Danry, Javier Hernandez, Andrew Wilson, Pattie Maes, Judith Amores
Researchers from Microsoft Research and MIT Media Lab developed a gaze-aware AI assistant that integrates egocentric sensing with multimodal LLM reasoning to interpret and adapt to users' cognitive needs during reading. The system led to a 7.4 percentage point increase in recall performance and was perceived by users as more accurate and personalized, while reducing the number of words spoken during interactions.
Key insights
- Directly interpreting gaze-projected egocentric video as a multimodal input for LLMs enables reasoning about user cognitive needs, moving beyond simple referential grounding or text-only analysis.
- Decomposing the inference of cognitive difficulty into distinct stages (gaze behavior observation and need state inference) allows for more nuanced and effective interpretation by LLMs.
- Integrating uncertainty-aware conversational strategies, such as hedging language and confirming user needs, is essential to manage the inherent ambiguity of gaze signals and maintain user agency.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.