Reproduce “Optimus-3: Dual-Router Aligned Mixture-of-Experts Agent with Dual-Granularity Reasoning-Aware Policy Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Weili Guan, Dongmei Jiang, Yaowei Wang, Liqiang Nie
Optimus-3, developed by researchers at Harbin Institute of Technology (Shenzhen Campus) and Pengcheng Laboratory, introduces a dual-router aligned Mixture-of-Experts agent that organically integrates fast System 1 actions with deliberative System 2 reasoning for open-ended tasks in Minecraft. The agent achieved a 35% success rate on the challenging Diamond Sword task and demonstrated improvements of up to 3.4 times on various System 2 reasoning tasks compared to prior state-of-the-art methods.
Key insights
- Synthesizing domain-specific reasoning data, rigorously grounded in environmental knowledge and feedback, effectively mitigates MLLM hallucinations and is crucial for developing robust System 2 capabilities in embodied agents.
- A dual-router MoE architecture, which decouples parameters for distinct tasks and dynamically adjusts inference depth per cognitive requirement, allows for the efficient and conflict-free integration of both fast reactive actions and deep deliberative reasoning within a unified model.
- Providing dual-granularity rewards that supervise both the intermediate reasoning process and the final outcome is essential for activating and refining System 2 reasoning, especially in visually grounded tasks where sparse outcome rewards lead to ungrounded reasoning and visual hallucinations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.