Reproduce “Evaluating Morphological Compositional Generalization in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mete Ismayilzada, Defne Circi, Jonne Sälevä, Hale Sirin, Abdullatif Köksal, Bhuwan Dhingra, Antoine Bosselut, Duygu Ataman and 1 more
Researchers investigated the morphological compositional generalization capabilities of Large Language Models (LLMs) using novel generative and discriminative tasks for Turkish and Finnish. The study found that current LLMs struggle to productively and systematically apply morphological rules to novel and complex linguistic inputs, demonstrating a notable performance gap compared to human capabilities.
Key insights
- LLMs demonstrate a consistent and significant inability to generalize morphologically, struggling with increased complexity and novel (OOD) word roots far more than humans.
- Error analysis indicated LLMs often exhibit "real word bias" and fail to adhere to explicit task constraints or apply morphological rules systematically, preferring sequential application over correct ordering.
- Providing correct morpheme order in the input prompt significantly boosts LLM performance, suggesting a reliance on input structure rather than inherent compositional derivation capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.