Reproduce “Human-like fleeting memory improves language learning but impairs reading time prediction in transformer language models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Abishek Thamma, Micha Heilbron
The authors test fleeting memory on 10M and 100M BabyLM sets, preserving a buffer of recent words while reducing the influence of older context. With the buffer, models improve validation loss and BLiMP accuracy; on 10M, accuracy rises from 68.97 to 70.01. Yet surprisal-based reading-time fit worsens, with Natural Stories ΔLL −176.29. The result applies only to tested scales.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.