Reproduce “Geometry is All You Need: A Unified Taxonomy of Matrix and Tensor Factorization for Compression of Generative Language Models” on OpenResearch

About this paper

Mingxue Xu, Sadia Sharmin, Danilo P. Mandic

Researchers at Imperial College London propose a unified taxonomy for matrix and tensor factorization methods used in generative language model compression, re-interpreting these techniques through the elementary geometric concept of a "subspace." This framework provides a common language for algebraic operations and existing compression approaches, aiming to improve accessibility and principled algorithm design in the field.

Key insights

• All matrix and tensor factorization techniques applied to LLM compression can be systematically understood as specific geometric operations (e.g., splitting, merging, projecting) on subspaces within a larger parameter space.
• A geometric analysis of LLM architectures suggests that MLP layers, through higher-dimensional linear projections, are correlated with the model's capacity for processing abstract concepts, while attention layers primarily facilitate fine-grained data distribution memorization.
• Current research in LLM compression predominantly focuses on splitting high-dimensional subspaces, indicating significant underexplored areas such as principled subspace construction, effective merging/discarding of less important subspaces, and strategic reallocation of dimensions.

Summary by alphaXiv.