Reproduce “Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected” on OpenResearch

About this paper

Yingtao Zhang, Diego Cerretti, Jialin Zhao, Wenjing Wu, Ziheng Liao, Umberto Michieli, Carlo Vittorio Cannistraci

A dynamic sparse training framework, CHTss, enables large language models and Transformers to achieve performance comparable to or surpassing fully connected networks at high sparsity, by addressing computational and topological limitations of prior brain-inspired methods. It demonstrates these capabilities across image classification, machine translation, and language modeling tasks, operating with as few as 5% of connections.

Key insights

• Optimizing brain-inspired dynamic sparse training by combining efficient link prediction and probabilistic topological evolution allows large models to perform comparably to or better than fully connected networks at extreme sparsity.
• The "soft rule" for link removal and regrowth is crucial for overcoming topological local minima and balancing the exploration and exploitation of network connectivity during training.
• Carefully designed sparse initialization methods (BRF) and smoother density decay curves (sigmoid) enhance training stability and performance in resource-efficient sparse ANNs.

Summary by alphaXiv.