Reproduce “HadaCore: Tensor Core Accelerated Hadamard Transform Kernel” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Krish Agarwal, Rishi Astra, Adnan Hoque, Mudhakar Srivatsa, Raghu Ganti, Less Wright, Sijia Chen

HadaCore introduces a specialized CUDA kernel that accelerates the Fast Walsh-Hadamard Transform by reformulating its algorithm to leverage NVIDIA GPU Tensor Cores. The approach achieves speedups of up to 3.6x over prior implementations while maintaining numerical accuracy, making rotation-based quantization more efficient for large language models.

Key insights

  • The Fast Walsh-Hadamard Transform's recursive butterfly structure can be reformulated to efficiently utilize Tensor Cores on modern NVIDIA GPUs.
  • Rotating weights and activations with Hadamard transforms before quantization effectively distributes outlier magnitudes, preserving accuracy in lower precision LLMs.
  • Careful CUDA kernel design, including work decomposition, memory access patterns, and register optimization, is crucial for maximizing hardware accelerator utilization in non-standard computations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.