Reproduce “Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kai Huang, Jian Zhang, Xiaofei Xie, Chunyang Chen
This research introduces GUIRepair, a cross-modal reasoning approach that enables Large Language Models to effectively diagnose and fix visual software issues by transforming information between visual and code modalities. The system achieved a 15.44% improvement over a base agentless approach, solving 157 instances on the SWE-bench M benchmark with GPT-4o and 175 instances with o4-mini, outperforming other state-of-the-art systems.
Key insights
- Transforming high-level visual information (screenshots) into executable code (Image2Code) significantly enhances LLM fault comprehension and localization for visual bugs.
- Providing visual feedback by converting code changes back into GUI renderings (Code2Image) offers a robust patch validation mechanism, crucial for visual issues where traditional test cases are insufficient.
- The combined application of Image2Code and Code2Image creates a bidirectional cross-modal reasoning pipeline that maximizes repair benefits for multimodal software issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.