Reproduce “Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On-Policy Self-Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shiyu Xuan, Zechao Li
A framework enables Graphical User Interface (GUI) visual grounding models to adapt and improve their performance on unseen interfaces after deployment without human-annotated ground truth. This approach, leveraging natural language reflections and self-distillation, consistently improved the grounding capabilities of base models by up to 7.7% on various benchmarks.
Key insights
- Rich, free-form natural language reflections can be effectively transformed into dense, token-level supervisory signals for continuous model improvement.
- The Contrastive Calibration method is crucial for robust learning during test-time adaptation, mitigating the negative effects of corrupted feedback from auto-regressive generation, especially during failed explorations.
- On-policy self-distillation can be successfully applied to enable test-time adaptation for GUI visual grounding, allowing models to evolve post-deployment without external ground truth.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.