Reproduce “Identifying User Goals from UI Trajectories” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Omri Berkovitch, Sapir Caduri, Noam Kahlon, Anatoly Efros, Avi Caciularu, Ido Dagan
Google Research introduces a new task to infer natural language user goals from observed sequences of UI interactions and screen images. Benchmarking state-of-the-art Large Multimodal Models like Gemini 1.5 Pro and GPT-4 against human performance on adapted datasets like Mind2Web and Android in the Wild reveals a notable gap, with models achieving around 58% accuracy compared to human 76-80% accuracy.
Key insights
- The task of inferring user goals from UI trajectories can be framed as the inverse of UI automation, allowing the repurposing of existing UI automation datasets for this new problem.
- State-of-the-art Large Multimodal Models (LMMs) like Gemini 1.5 Pro and GPT-4 exhibit limitations in visual screen understanding and nuanced contextual reasoning within UI environments, struggling with precise detail extraction and avoiding irrelevant information.
- The research highlights a substantial performance gap between current LMMs and human capabilities in accurately identifying specific user intents from UI interaction traces.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.