Reproduce “Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models” on OpenResearch

About this paper

Zichen Miao, Wei Chen, Qiang Qiu

This paper introduces Coeff-Tuning, a parameter-efficient fine-tuning method that reformulates multi-head attention as a graph convolutional filter subspace. By tuning a minimal set of unconstrained subspace coefficients, the method enhances attention expressiveness, allowing output embeddings to occupy a larger feature space, and consistently improves performance across various large Transformer models and tasks with negligible parameter overhead.

Key insights

• Standard multi-head attention maps, due to the softmax function, inherently limit output token embeddings to a convex hull of value embeddings, restricting expressiveness.
• Multi-head attention can be re-conceptualized as forming a 'graph convolutional filter subspace,' where each head is a filter 'atom'.
• Tuning unconstrained linear combinations of these pre-trained filter atoms with minimal coefficients allows the output feature space to expand beyond the original convex hull.

Summary by alphaXiv.