Reproduce “Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianpeng Bu, Xin Liu, Qihua Chen, Hao Jiang, Shurui Li, Hongtao Duan, Lu Jiang, Lulu Hu and 2 more
The authors introduce GUI-RobustEval and Robustness-driven Trajectory Synthesis, or RoTS, to test and train GUI agents to recognize policy-induced errors and recover. GUI-RobustEval contains 1,216 test cases; RoTS expands rollout branches through FDE and EIR. On 369 OSWorld-Verified tasks, RoTS-32B reaches 47.4% success and 33.8% All-Pass@4 across four runs, versus 34.1% and 15.5% for OpenCUA-32B, with evidence limited to desktop tasks.
Key insights
- Policy-induced errors in real-world GUI agent execution are often compositional and require long-horizon recovery, a challenge poorly addressed by current benchmarks and training methodologies.
- A systematic online data synthesis framework, like RoTS, can effectively generate high-quality training data that covers diverse and long-horizon policy-induced errors, bridging the existing data gaps.
- Improved long-horizon error awareness and recovery capabilities directly correlate with enhanced overall robustness and task success rates, making agents more reliable for practical applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.