Reproduce “Multimodal Few-Shot Learning with Frozen Language Models” on OpenResearch

About this paper

Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, Felix Hill

The DeepMind paper demonstrates how to extend the few-shot learning capabilities of large, pre-trained language models to visual tasks. The developed system, named Frozen, effectively achieves rapid task adaptation, encyclopedic knowledge retrieval, and fast concept binding in multimodal settings by training only a vision encoder while keeping the language model's parameters fixed.

Key insights

• The few-shot learning, encyclopedic knowledge retrieval, and fast concept binding capabilities observed in text-only LLMs can successfully transfer and extend to multimodal tasks when appropriately grounded with visual information.
• Keeping the large language model frozen is essential for preserving its pre-trained linguistic knowledge and few-shot generalization, offering a modular and efficient approach to multimodal integration.
• Training only a smaller, modality-specific encoder through a frozen, powerful language model provides a scalable method to ground LLMs in new sensory inputs without significant computational overhead or catastrophic forgetting.

Summary by alphaXiv.