Reproduce “CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Koustuv Sinha, Shagun Sodhani, Jin Dong, Joelle Pineau, William L. Hamilton
CLUTRR is a diagnostic benchmark designed to evaluate inductive reasoning from text in Natural Language Understanding (NLU) systems. The benchmark generates short stories requiring kinship inference, revealing a substantial performance gap where structured graph-based models outperform text-based NLU models in systematic generalization and robustness to noise, highlighting challenges in extracting logical structures from diverse natural language.
Key insights
- A significant performance gap exists between models processing unstructured text (NLU systems) and those operating on structured symbolic inputs (Graph Attention Networks) when performing inductive logical reasoning.
- The primary limitation for current text-based NLU models in this task is robustly extracting and mapping diverse natural language narratives to their underlying logical facts, rather than an inherent inability to perform the logical inference itself.
- Models struggle with systematic generalization to more complex reasoning chains not explicitly seen during training, and text-based NLU models show limited ability to learn robust reasoning strategies from noisy data compared to structured models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.