Reproduce “An All-MLP Sequence Modeling Architecture That Excels at Copying” on OpenResearch

About this paper

Chenwei Cui, Zehao Yan, Gedeon Muhawenayo, Hannah Kerner

The Causal Relation Network (CausalRN), an all-MLP sequence modeling architecture, demonstrates performance on the copying task comparable to Transformers by incorporating exponential activation and pre-activation normalization. This architecture achieves 99% accuracy faster than Transformers across various string lengths, challenging the exclusivity of attention mechanisms for robust in-context retrieval.

Key insights

• An all-MLP architecture can achieve Transformer-level performance on complex in-context retrieval tasks when equipped with specific architectural innovations like exponential activation and pre-activation normalization.
• The exponential activation function enables a theoretical linearization of the CausalRN, paving the way for highly efficient sequence models with reduced computational complexity.
• Pre-activation normalization is critical for establishing and maintaining an 'irreducible memory pool,' which is a functional analogue to a Transformer's KV cache and vital for robust retrieval capabilities.

Summary by alphaXiv.