Reproduce “Event-Driven Language Models with Sparse Neural Activity for Neuromorphic Hardware” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Simon Richter, Ruhai Lin, Jason Yik, Taylor Kergan, Rui-Jie Zhu, Farshad Moradi, Jason Eshraghian
A method was developed to induce high activation sparsity in heavily quantized linear-attention language models, specifically designed for event-driven neuromorphic hardware. This approach achieves substantial improvements in projected throughput and energy efficiency for large language models on edge devices by leveraging inherent hardware capabilities.
Key insights
- Activation sparsity sensitivity varies significantly across different layers and projection types within the language model, necessitating a fine-grained, per-projection sparsification approach for optimal performance retention.
- The two-sided ReLU mechanism, with a learnable threshold, effectively creates high sparsity by nullifying activations within a specific range while preserving the magnitude of crucial outlier activations.
- Event-driven neuromorphic hardware inherently capitalizes on unstructured sparsity at both intra-chip (skipping MAC operations) and inter-chip (reducing communication overhead) levels, translating sparsity directly into efficiency gains that conventional GPUs cannot achieve.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.