Reproduce “MemTrain: Self-Supervised Context Memory Training” on OpenResearch

About this paper

Ziheng Li, Xingrun Xing, Haoqing Wang, Zhi-Hong Deng, Yehui Tang

MemTrain, a self-supervised training framework, endows large language models with general context memory capabilities by leveraging unlabeled corpora. This approach provides a strong initialization for downstream tasks, yielding substantial improvements in memory-intensive reasoning across diverse benchmarks and context lengths.

Key insights

• The combination of outcome-level (end-to-end masked reconstruction) and process-level (intermediate memory recall) supervision is essential for robust memory learning in long-horizon interactions.
• Self-supervised memory pre-training significantly elevates the performance ceiling for subsequent task-specific fine-tuning, rather than just accelerating it.
• Generalizable memory capabilities can be effectively acquired from readily available, unlabeled text, reducing dependency on expensive annotated datasets for memory-intensive tasks.

Summary by alphaXiv.