Reproduce “GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang and 12 more
GUI-WORLD introduces a video benchmark and dataset of 12,379 GUI videos across six diverse scenarios to enhance multimodal GUI understanding, focusing on dynamic and sequential content. An exploratory GUI-VID model fine-tuned on this dataset achieves an average performance improvement of 30% over its baseline in various GUI tasks.
Key insights
- Commercial vision LLMs (e.g., GPT-4o, GPT-4V) outperform open-source video LLMs in zero-shot GUI understanding, but even they struggle with detailed descriptions of entire videos and fine-grained static content.
- Dynamic GUI tasks and temporal understanding remain significant challenges for current MLLMs, highlighting the need for improved temporal reasoning and visual perception capabilities.
- Enhancing vision perception through higher resolution and more keyframes, alongside integrating detailed textual information, significantly improves MLLM performance in dynamic GUI tasks.
- Model-based keyframe selection strategies, particularly those adapted from embodied AI, show promise in automating the labor-intensive manual keyframe extraction process for dataset scaling.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.