Reproduce “HyperMixer: An MLP-based Low Cost Alternative to Transformers” on OpenResearch

About this paper

Florian Mai, Arnaud Pannatier, Fabio Fehr, Haolin Chen, Francois Marelli, Francois Fleuret, James Henderson

HyperMixer introduces an MLP-based neural network architecture that offers a cost-effective alternative to Transformers for natural language understanding. This model achieves competitive or superior performance across several GLUE benchmark tasks while significantly reducing computational expense, data requirements, and tuning effort by using hypernetworks to dynamically generate an attention-like token mixing function.

Key insights

• MLP-based architectures can achieve performance comparable to or superior to Transformers in NLU tasks when designed with appropriate inductive biases like position invariance and systematicity.
• Attention-like token interaction patterns, which enable variable binding and systematicity, can be learned efficiently through dynamic weight generation by hypernetworks, without requiring explicit attention mechanisms.
• Significant reductions in computational cost, data requirements, and hyperparameter tuning effort are achievable for high-performing NLP models, aligning with the goals of 'Green AI'.

Summary by alphaXiv.