Reproduce “JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He and 4 more
An agent for open-world environments, JARVI S-1, leverages a memory-augmented multimodal language model to achieve robust multi-task performance and continuous self-improvement in Minecraft. It demonstrates up to 5x higher reliability on complex tasks like `ObtainDiamondPickaxe` compared to prior state-of-the-art models.
Key insights
- Integrating multimodal perception (e.g., visual observations processed by MineCLIP) with LLMs is crucial for situation-aware and robust planning in dynamic environments.
- A multimodal memory system enables efficient "in-context life-long learning," allowing agents to continuously improve planning skills on long-horizon tasks without gradient updates.
- Autonomous task generation via "self-instruct" and distributed experience collection can significantly accelerate the acquisition of diverse skills and complex behaviors.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.