Reproduce “UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhengxi Lu, Fei Tang, Guangyi Liu, Kaitao Song, Xu Tan, Jin Ma, Wenqi Zhang, Weiming Lu and 3 more
A collaborative framework, UI-Copilot, enhances long-horizon GUI automation by integrating specialized tools on demand and optimizing policies with Tool-Integrated Policy Optimization (TIPO). The framework demonstrated state-of-the-art performance on the MemGUI-Bench, achieving a 16.4% pass@1 accuracy, and improved generalization across various GUI benchmarks.
Key insights
- Decoupling detailed reasoning and historical context from the main agent's context window significantly mitigates memory degradation and progress confusion in long-horizon tasks.
- Selectively invoking lightweight, specialized tools only when needed improves efficiency and reduces inference costs compared to always-on external modules.
- A reinforcement learning strategy that explicitly separates the learning of tool calls from action generation, leveraging on-policy history for action prediction, leads to robust and efficient agent behavior.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.