Reproduce “Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAM” on OpenResearch

About this paper

Bingbing Li, Geng Yuan, Zigeng Wang, Shaoyi Huang, Hongwu Peng, Payman Behnam, Wujie Wen, Hang Liu and 1 more

Researchers from the University of Connecticut and collaborators developed a fault tolerance strategy for Transformer-based language models on ReRAM that incurs zero additional storage cost. The method integrates differentiable structured pruning with an MSB duplication and voting strategy, enhancing accuracy preservation by nearly 10% under a 0.05% failure rate compared to a baseline without voting.

Key insights

• Deep Neural Networks inherently possess sufficient redundancy that can be leveraged through pruning to create 'zero-space' for embedding fault-tolerant duplicates without increasing physical footprint.
• More than 99% of MSBs in quantized BERT weights are zero, making them highly vulnerable to Stuck-At-1 (SA1) faults and justifying a specialized bit-flipping protection strategy.
• A synergistic integration of model compression and hardware-aware fault tolerance offers a practical solution to make large AI models robust and efficient on emerging memory technologies.

Summary by alphaXiv.