Reproduce “MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Control” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, Jing Shao

MineDreamer enables generalist embodied agents to reliably follow diverse, abstract, and sequential natural language instructions for low-level control in open-world environments like Minecraft by introducing a Chain-of-Imagination mechanism, significantly outperforming prior methods on various tasks including complex, multi-step objectives.

Key insights

  • Iterative, situation-aware 'imagination' of the next immediate step (Chain-of-Imagination) is crucial for steady and reliable instruction following in dynamic, open-world environments.
  • Multimodal Large Language Models (MLLMs) can be effectively integrated with diffusion models to generate high-quality, instruction-aligned, and physically consistent visual imaginations for guiding embodied agents.
  • Transforming abstract instructions and imaginations into precise latent visual prompts, combined with pre-trained low-level controllers, unlocks instruction-following capabilities in generalist foundation models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.