Reproduce “LECAT: Measuring LLM Category Learning from Pretraining Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jimin Mun, Maarten Sap, Yanai Elazar
LECAT measures how language models apply category-feature rules across contexts and how training-data expression affects that behavior. It evaluates feature-likelihood orderings for memorization, generalization, and contextual flexibility. In controlled 68-million-parameter models trained on TinyStories with two fictional relations, a mixture scored 0.51 overall versus 0.47 for repetition alone, but weakening-context distinctions remained difficult, passing at most 0.13.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.