Reproduce “Large Language Models Implicitly Learn to See and Hear Just By Reading” on OpenResearch

About this paper

Prateek Verma, Mert Pilanci

Large Language Models, pre-trained exclusively on text, implicitly acquire the ability to understand and classify non-textual data. Researchers demonstrated that a GPT-2 model, fine-tuned with LoRA, can directly process and classify images and audio, achieving competitive performance with significantly fewer trainable parameters than models trained from scratch.

Key insights

• Text-only LLMs develop internal representations and 'circuits' that are robust and modality-agnostic, allowing them to interpret diverse sensory data beyond their training domain.
• Fine-tuning even a small subset of parameters within a pre-trained text LLM backbone (via LoRA) is more effective for multimodal adaptation than using a frozen backbone.
• The scaling laws observed in text LLMs, where larger models exhibit improved performance, extend to their latent multimodal capabilities, showing increased accuracy and faster convergence for non-textual tasks.

Summary by alphaXiv.