Reproduce “Matmul or No Matmul in the Era of 1-bit LLMs” on OpenResearch

About this paper

Jinendra Malekar, Mohammed E. Elbtity, Ramtin Zand

Researchers from the University of South Carolina quantified the actual computational and memory benefits of 1-bit Large Language Models, revealing that the efficacy of 1-bit quantization varies significantly with model size, sequence length, and deployment hardware (cloud vs. edge). The work provides a data-driven roadmap for future research and hardware design, directing efforts towards the most impactful optimization areas.

Key insights

• The fraction of MatMul-free operations (projection layers) varies significantly (23% to 98% for computation, 59% to 99.8% for memory), increasing with model size but decreasing with sequence length.
• On cloud setups, improving attention heads yields greater overall performance gains for smaller LLMs, while 1-bit optimization of projection layers provides substantial benefits for larger LLMs.
• For edge deployments, optimizing projection layers consistently leads to significantly greater overall improvements across all model sizes, with minimal gains from attention layer enhancements.

Summary by alphaXiv.