Reproduce “Entity tracking emerges in sub-billion parameter language models and exceeds human performance in naturalistic narratives” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Karolina Drożdż, Micha Heilbron
The authors test whether models track object locations as events unfold in short narratives. They compare people and models using implicit probabilities of correct versus incorrect continuations. Pythia reaches human-level accuracy on the implicit task at 410M parameters, while accuracy rises from 53.5% at 70M to 89.6% at 12B. This finding concerns the task, not general comprehension or internal processes.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.