Reproduce “KARIPAP: Quantum-Inspired Tensor Network Compression of Large Language Models Using Infinite Projected Entangled Pair States and Tensor Renormalization Group” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Azree Nazri
KARIPAP introduces a quantum-inspired tensor network framework, utilizing Infinite Projected Entangled Pair States (iPEPS) and the Tensor Renormalization Group (TRG), to compress Large Language Models like LLaMA-2 7B. This approach achieves substantial reductions in memory, training time, and inference latency while preserving model accuracy, offering a more sustainable and efficient method for deploying advanced AI.
Key insights
- Deeper transformer layers in LLMs exhibit redundant entanglement patterns, confirming that these models are over-parameterized and possess low-dimensional entanglement manifolds.
- Quantum-inspired tensor networks, particularly iPEPS, provide a mathematically rigorous and correlation-aware framework to efficiently model the intrinsic entanglement geometry within LLM parameter spaces.
- The TRG algorithm effectively approximates complex tensor network contractions, enabling the practical application of 2D entanglement structures for LLM compression.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.