Reproduce “CognitiveDog: Large Multimodal Model Based System to Translate Vision and Language into Action of Quadruped Robot” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Artem Lykov, Mikhail Litvinov, Mikhail Konenkov, Rinat Prochii, Nikita Burtsev, Ali Alridha Abdulkarim, Artem Bazhenov, Vladimir Berman and 1 more

Skoltech researchers developed CognitiveDog, a multi-modal, multi-agent system on a quadruped robot that translates natural language and visual input into physical actions. The system achieves superior generalization in diverse environments and exceptional reasoning capabilities compared to larger state-of-the-art models like RT-2, while utilizing approximately four times fewer parameters.

Key insights

  • An "inner monologue" architecture, where distinct transformer models communicate in natural language, enables complex reasoning, adaptive plan generation, and environmental information gathering.
  • Leveraging smaller, highly capable open-source models (Mistral 7B, MiniGPT4-v2) can achieve superior performance and efficiency compared to larger models in specific embodied AI tasks.
  • The adaptability of pre-trained visual-language models like MiniGPT4-v2 is crucial for generalization across diverse environments, even without task-specific training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.