Reproduce “Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, Liqiang Nie
Optimus-2 proposes a Goal-Observation-Action Conditioned Policy (GOAP) for Minecraft agents, which explicitly models observation-action causal relationships and long-term behavioral sequences while integrating an MLLM for language understanding. This architecture, trained on a new automatically generated Minecraft Goal-Observation-Action (MGOA) dataset, enhances performance across atomic, long-horizon, and open-ended tasks in the Minecraft environment.
Key insights
- Explicitly modeling the causal relationship between observations and previous actions significantly enhances policy performance and yields discriminative behavior representations.
- Dynamically aggregating long-term observation-action sequences through a history mechanism allows the policy to learn from past contexts and make coherent predictions.
- Integrating a Multimodal Large Language Model (MLLM) directly into the policy's backbone improves its comprehension and alignment with open-ended natural language sub-goals.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.