Reproduce “Computer-Using World Model” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiming Guan, Rui Yu, John Zhang, Lu Wang, Chaoyun Zhang, Liqun Li, Bo Qiao, Si Qin and 10 more
The Computer-Using World Model (CUWM) enables large language model agents to simulate the effects of their actions within complex desktop graphical user interface environments, specifically Microsoft Office applications. This two-stage model facilitates counterfactual reasoning, improving agent decision-making and task completion rates by allowing agents to "think before acting" without real-world execution.
Key insights
- Factorizing UI dynamics into a semantic textual transition and a visual state realization provides an interpretable and effective approach to modeling complex desktop GUI changes.
- Utilizing reinforcement learning with an LLM-as-a-Judge reward function effectively aligns the textual state transition model with decision-critical UI structures for agents.
- World models significantly enhance agent decision-making by enabling counterfactual exploration and test-time action search, thereby mitigating risks and costs associated with live execution in desktop applications.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.