Reproduce “Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols and 2 more
Ferret-UI 2, developed by researchers at Apple, presents a generalist multimodal large language model capable of universal user interface understanding and single-step, user-centered interaction across iPhone, Android, iPad, Webpage, and AppleTV platforms. It achieved an 89.73 GPT-4o score on advanced tasks and consistently outperformed prior models and plain GPT-4o on various benchmarks.
Key insights
- The combination of the adaptive N-gridding architecture and meticulously curated, high-quality, visually-grounded training data significantly contributes to robust UI understanding and performance gains.
- GPT-4o with set-of-mark visual prompting is crucial for generating superior training data that enhances the model's spatial understanding and ability to interpret complex user intents.
- While some mobile platforms show natural cross-transferability, truly universal UI understanding requires comprehensive data coverage across diverse UI types due to platform-specific disparities like orientation and content distribution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.