Reproduce “Practice in Text, Act in Pixels: Correcting Where Perception Fails” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhongling Xu, Hairong Wang

The authors align a vision-language reader using about 2,000 screenshots paired with interface-element lists, leaving the policy unchanged. This targets recurring reader errors: 73.2% of misread elements recur, versus 30.9% under independent errors. On new instances of practiced MobileGym tasks, the aligned reader achieves 80.9% success, versus 55.6% for direct screenshot training; this does not establish real-device task performance.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.