Reproduce “LLM Agents as Computational Typologists” on OpenResearch

About this paper

Changbing Yang, Christopher Hammerly, Freda Shi, Jian Zhu

AUTOTYPOLOGIST applies a tool-using, evidence-tracking LLM workflow to two typological tasks: coding Grambank features from reference grammars and testing crosslinguistic hypotheses. Across 25 grammars and 195 features, grammar-based analysis reached up to 70.7% weighted F1, while the agent also distinguished support from counterevidence for standard and adversarial universals, though subtle grammatical interpretations still required expert review.

Key insights

• Reference-grammar prose was generally more useful than IGT alone: Qwen3.6-27B reached 70.7% weighted F1 with grammar access versus 63.8% using only IGT, while combining both sources did not consistently improve performance.
• Longer reasoning helped macro F1 in some IGT-only experiments but did not reliably improve weighted F1, indicating that additional computation cannot by itself resolve retrieval and linguistic-interpretation errors.
• The agent can separate supporting, opposing, irrelevant, and insufficient evidence, but it remains vulnerable to distinctions involving grammatical function, construction type, exponence, and terminology mismatches.

Summary by alphaXiv.