Reproduce “Inductive Linguistic Reasoning with Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Raghav Ramji, Keshav Ramji
A two-stage analogical prompting method enables large language models to perform sophisticated inductive linguistic reasoning on low-resource languages. The approach significantly improves performance on linguistic puzzles, with Llama-3.1-405B-Instruct achieving over 70% Exact Match on the modeLing dataset.
Key insights
- Frontier large language models exhibit a strong parametric understanding of language families, often inferring them more effectively than using provided oracle labels.
- The process of linguistic reasoning in LLMs can be effectively decomposed into a stage of analogical exemplar generation (e.g., by a specialized multilingual model) and a subsequent stage of rule deduction and application (by a generalist model).
- Linguistics Olympiad-style puzzles serve as robust benchmarks for inductive reasoning, revealing nuanced capabilities and limitations in LLMs, including their handling of language isolates and uncertainty-based refusal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.