Reproduce “Multi-Turn Reasoning When Context Arrives in Pieces: Scalable Sharding and Memory-Augmented RL” on OpenResearch

About this paper

Shu Tong Luo, Wenqin Liu, Rui Liu, Mingming Gong, Jiaxian Guo

A memory-augmented reinforcement learning framework from The University of Melbourne and The University of Tokyo mitigates the "Lost in Conversation" (LiC) phenomenon in LLMs by training models on scalably sharded data, resulting in 15.2 to 35.6 percentage point accuracy gains compared to full-history baselines and strong zero-shot generalization. This approach teaches models to maintain a compact, bounded natural-language memory, outperforming methods that rely on full chat history.

Key insights

• Training LLMs with a constrained, rolling natural-language memory buffer induces a more robust and effective incremental reasoning capability than merely processing increasingly long full conversation histories.
• Scalably generated multi-turn fragmented data, particularly when interleaved with noise, effectively builds both problem-solving skills and the ability to filter irrelevant information, crucial for real-world contexts.
• The memory-augmented RL approach fosters a domain-agnostic incremental reasoning capability, enabling zero-shot generalization to harder mathematical tasks and diverse long-context QA benchmarks.

Summary by alphaXiv.