Reproduce “Learning to Transduce with Unbounded Memory” on OpenResearch

About this paper

Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, Phil Blunsom

Researchers from Google DeepMind and Oxford University developed training-free, continuously differentiable memory modules—Neural Stack, Queue, and DeQue—to augment LSTMs for sequence transduction tasks. These memory-enhanced LSTMs demonstrate superior generalization to input sequences twice as long as training data, achieve perfect accuracy on complex algorithmic and linguistic tasks, and converge faster than conventional Deep LSTMs.

Key insights

• Providing neural networks with explicit, unbounded memory structures enables them to learn the underlying procedural algorithms of a task, rather than just memorizing input-output mappings.
• Careful design of differentiable memory operations, including handling non-differentiable points and decoupling memory dynamics from the controller, is crucial for effective end-to-end training.
• A continuous DeQue offers versatile memory access, allowing the network to adapt to tasks requiring both stack-like and queue-like behaviors, leading to robust performance.

Summary by alphaXiv.