Reproduce “ScreenAI: A Vision-Language Model for UI and Infographics Understanding” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin and 2 more
Google DeepMind developed ScreenAI, a Vision-Language Model designed for comprehensive understanding of both user interfaces and infographics. This model employs a unified textual representation and LLM-generated training data to achieve state-of-the-art performance, including 87.4 Acc@IoU=0.1 on MoTIF and 77.1 ANLS on Multipage DocVQA.
Key insights
- A unified Vision-Language Model can effectively understand both UIs and infographics by leveraging their shared visual language and design principles, leading to positive knowledge transfer across domains.
- Automatically generating vast and diverse training data using LLMs, guided by structured representations, provides a scalable solution to overcome manual annotation bottlenecks and enhance model capabilities.
- Model performance consistently improves with increasing parameter size, especially for complex visual-text and arithmetic reasoning tasks, indicating the scalability of the proposed architecture.
- Flexible patching mechanisms are essential for adapting VLMs to the varied aspect ratios and resolutions commonly found in screen-based digital content.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.