Reproduce “Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity” on OpenResearch

About this paper

Pranava Madhyastha, Dagmar Adamcova

Research at City, University of London and The Alan Turing Institute explored how integrating architectural constraints inspired by human working memory into Transformer models influences language learning. The study found these constraints enhanced grammatical competence and alignment with human psychometric data, particularly when trained on limited datasets, while also promoting more structured internal representations.

Key insights

• Architectural biases inspired by human working memory serve as effective inductive biases, particularly under data scarcity, leading to more robust linguistic representations.
• The language modeling objective and human comprehension may diverge as models are trained on increasingly large datasets, with psychometric alignment degrading at higher data volumes.
• Explicit cognitive constraints encourage a structured division of labor among attention heads and facilitate the emergence of more explicit and linearly recoverable syntactic representations earlier in the network.

Summary by alphaXiv.