Reproduce “AndroidGen: Building an Android Language Agent under Data Scarcity” on OpenResearch

About this paper

Hanyu Lai, Junjie Gao, Xiao Liu, Yifan Xu, Shudan Zhang, Yuxiao Dong, Jie Tang

ANDROIDGEN introduces a framework for building Android language agents that addresses data scarcity by autonomously generating and augmenting high-quality interaction trajectories. The system enables the training of robust open-source large language models, achieving an average success rate of 46.8% on the AndroidWorld benchmark with a GPT-4o variant and 35.3% with a fine-tuned Llama-3-70B.

Key insights

• LLMs can effectively self-generate high-quality training data for complex interactive tasks, overcoming the data scarcity bottleneck when guided by robust evaluation and augmentation processes.
• A modular agent design significantly improves performance by addressing specific weaknesses of LLMs in agentic tasks, such as planning robustness, operational accuracy, and leveraging past experience.
• Fine-grained step-by-step evaluation, as provided by StepCritic, is crucial for both filtering generated data and enabling effective data augmentation, leading to more targeted model improvements.

Summary by alphaXiv.