Reproduce “Scavenging Hyena: Distilling Transformers into Long Convolution Models” on OpenResearch

About this paper

Tokiniaina Raharison Ralambomihanta, Shahrad Mohammadzadeh, Mohammad Sami Nur Islam, Wassim Jabbour, Laurence Liang

Scavenging Hyena introduces a method for distilling knowledge from pre-trained Transformer models into computationally efficient Hyena-based long convolution architectures, enabling these student models to achieve comparable or superior language understanding capabilities with reduced training costs. This approach produced Hyena models with perplexity scores significantly lower than those trained from scratch and demonstrated improved accuracy on specific tasks like WSC.

Key insights

• Knowledge distillation can effectively transfer complex language modeling capabilities between fundamentally different neural network architectures (Transformer to Hyena).
• Progressive knowledge transfer, especially when combined with CE fine-tuning, offers a more efficient initialization strategy for novel architectures compared to training from scratch.
• Sub-quadratic Hyena models, when properly distilled, can achieve language understanding performance competitive with and occasionally exceeding that of attention-based Transformers on specific tasks.

Summary by alphaXiv.