Reproduce “Experience Grounds Language” on OpenResearch

About this paper

Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou and 4 more

This position paper argues that deep language understanding requires grounding in real-world experience beyond text-only learning, introducing the "World Scope" framework to integrate multimodal perception, embodied interaction, and social context. It posits this approach is necessary to overcome limitations of current large language models and enable more robust, human-like communication capabilities.

Key insights

• Language meaning is fundamentally derived from its use by people in the physical and social world, not merely from word distributions in text.
• Moving beyond text-only learning to include perception, embodiment, and social interaction provides essential semantic axioms, world knowledge, and addresses issues like reporting bias.
• Achieving human-like language understanding necessitates models capable of active interaction, causal reasoning, and inferring mental states, which are only possible through higher World Scopes.

Summary by alphaXiv.