Reproduce “FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qinglong Yang, Haoming Li, Haotian Zhao, Xiaokai Yan, Jingtao Ding, Fengli Xu, Yong Li
The FingerTip 20K benchmark and dataset are introduced to evaluate mobile LLM agents on proactive task suggestion and personalized task execution. Experiments reveal current state-of-the-art models struggle with these complex, user-centric tasks but show considerable performance improvements when fine-tuned on the real-world, longitudinal interaction data.
Key insights
- Proactive and personalized mobile assistance presents a substantial challenge for current MLLMs and specialized GUI-control agents, indicating a significant gap with human performance.
- Integrating user-related contextual information, such as historical intents, time, and partial screenshots, is crucial for enhancing proactive intent prediction.
- Fine-tuning models on the FingerTip 20K dataset significantly improves both proactive intent prediction and the personalization of action sequences during task execution.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.