Reproduce “VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model” on OpenResearch

About this paper

Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang and 2 more

Researchers from Microsoft and collaborating universities developed VEM, an environment-free reinforcement learning framework for training GUI agents. VEM utilizes LLM-guided annotations to train a Value Environment Model, which then guides an offline policy optimization process, achieving state-of-the-art task success rates on the Android-in-the-Wild benchmark, surpassing online-trained baselines while using one-third of the data.

Key insights

• Large Vision-Language Models can effectively distill human-like priors about GUI interaction outcomes through semantic annotation, enabling robust value estimation without direct environmental feedback.
• Decoupling the value function by pretraining and freezing the VEM significantly enhances training stability and efficiency, mitigating compounding errors and gradient variance in policy optimization.
• Environment-free training, guided by semantic value estimation, can achieve and surpass the performance of costly online interaction methods for GUI agents with greater sample efficiency and robustness to dynamic environments.

Summary by alphaXiv.