Reproduce “MintAct: A Unified Visual Agent for Digital Environments” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mingfei Gao, Rui Tian, Haiming Gang, Bohan Zhai, Le Zhang, Yuanzheng Gong, Di Feng, Ege Özsoy and 6 more
MintAct combines grounding, navigation, and visual tool use in one vision-language model family spanning mobile, desktop, web, and tool-based tasks. It uses shared screenshots and pixel coordinates for interface work, domain-specific prompts, and staged supervised and reinforcement learning. MintAct-8B scores 48.9 on OSWorld-Verified versus 46.1 for EvoCUA-8B, while joint reinforcement learning covers only mobile and desktop.
Key insights
- High-resolution single-step training is important for precise grounding, whereas low-resolution multi-step training provides the context and temporal behavior needed for navigation and tool use; combining them avoids sacrificing grounding quality.
- A single compact model can approach or exceed specialist performance across several digital-interaction domains when the data mixture, action conditioning, and optimization process are designed together.
- Joint mobile-desktop RL transfers to web navigation even without web data in that stage, but provides weaker benefits for visual tool use, whose interaction pattern is more distant from GUI control.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.