Reproduce “GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Saelyne Yang, Jaesang Yu, Yi-Hao Peng, Kevin Qinghong Lin, Jae Won Cho, Yale Song, Juho Kim
GUIDE is a new benchmark for multimodal large language models (MLLMs) designed to evaluate their ability to understand and assist users in open-ended graphical user interface (GUI) tasks. It provides a dataset of novice user interactions and an evaluation framework focusing on behavior state detection, intent prediction, and help prediction, demonstrating that current MLLMs benefit significantly from structured contextual information for effective assistance.
Key insights
- State-of-the-art MLLMs struggle to accurately infer nuanced user behavior states (e.g., frustration) and provide context-aware help from visual cues alone.
- Incorporating structured user context (behavior state, inferred intent) substantially and consistently improves MLLM performance in predicting help needs and content.
- Performance generally improves with increased temporal context, indicating that accumulating visual information over time is beneficial for user understanding tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.