Reproduce “BitROM: Weight Reload-Free CiROM Architecture Towards Billion-Parameter 1.58-bit LLM Inference” on OpenResearch

About this paper

Wenlun Zhang, Xinyu Li, Shimpei Ando, Kentaro Yoshioka

BitROM presents a Compute-in-Read-Only-Memory architecture optimized for billion-parameter 1.58-bit LLM inference on edge devices, achieving 10x area efficiency and 20.8 TOPS/W. The design integrates on-die KV-cache management and Low-Rank Adaptation for efficient auto-regressive decoding and task flexibility.

Key insights

• Co-designing a digital CiROM architecture with the specific properties of 1.58-bit ternary quantized LLMs, including simplified operations and weight sparsity, enables significant gains in area efficiency and energy-efficient computation.
• Buffering a portion of the KV-cache on-die using eDRAM, and leveraging its inherent read-refresh mechanism, effectively mitigates the external memory bottleneck for auto-regressive LLM inference in edge scenarios.
• Targeted application of Low-Rank Adaptation (LoRA) to specific Transformer layers with low-bit quantization provides an effective and low-overhead mechanism for task flexibility and domain adaptation in fixed-weight CiROM designs for LLMs.

Summary by alphaXiv.