Reproduce “Relational reasoning and inductive bias in transformers and large language models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jesse Geerts, Andrew Liu, Stephanie Chan, Claudia Clopath, Kimberly Stachenfeld
The authors test how learning regime affects transitive inference in controlled transformers and language-model prompts. Small IWL models generalize from adjacent training pairs to untrained non-adjacent pairs, while standard ICL models do not unless trained on such examples. Linear-regression pretraining enables ICL generalization. Across seven models, number-line prompts outperform circle prompts in six, but do not identify internal reasoning mechanisms.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.