Reproduce “OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Qiushi Sun, Kanzhi Cheng, Yian Wang, Bowen Yang, Hang Yan, Liheng Chen, Fangzhi Xu, Zichen Ding and 15 more
This research systematically evaluates the reliability of Vision-Language Models (VLMs) as judges for cross-platform computer-using agent (CUA) trajectories, revealing a pervasive leniency bias where frontier models achieve only ~90% accuracy and open-source alternatives perform significantly worse. In response, a standardized benchmark (OSReward) and a cost-effective, open-source reward model (OS-Shepherd) are introduced, with OS-Shepherd closing the reliability gap while reducing costs by 30-60 times.
Key insights
- VLM judges exhibit a pervasive 'leniency bias,' frequently accepting incomplete or failed tasks as successes, driven by over-reliance on the agent's self-narrative rather than strict environmental verification.
- Judge performance significantly degrades on genuinely difficult trajectories (OSReward-Hard), with even top models dropping below 70% accuracy, and is weakly influenced by variations in visual input like the number of screenshots or presence of click markers.
- Removing the agent's textual thought and action history severely impacts VLM judge accuracy, highlighting their heavy dependence on this narrative, while increasing 'thinking' effort or ensembling offers only marginal aggregate gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.