Reproduce “Data-Efficient Adaptation of LLMs via Attention Head Reweighting” on OpenResearch

About this paper

Tuomas Oikarinen, Zixiao Chen, Charlotte Siska, Tsui-Wei Weng, Chandan Singh, Jianfeng Gao

Researchers from UC San Diego and Microsoft Research introduce Attention Head Reweighting (AHR), an ultra-parameter-efficient method for adapting large language models to new text-classification tasks in data-scarce environments. AHR achieves average absolute accuracy improvements of 2-4 percentage points over PEFT baselines in extremely low-data regimes, while using 200-1000 times fewer trainable parameters.

Key insights

• Extremely constrained adaptation by reweighting only a minimal number of attention heads prevents overfitting in data-scarce settings, leading to better performance than methods with larger parameter spaces.
• Attention heads exhibit functional specialization, with some heads being generally useful for in-context learning across tasks and others specializing for specific tasks, which can be identified through their learned reweighting factors.
• Combining finetuning with in-context learning principles via In-Context Finetuning (IC-FT) significantly boosts the performance of all adaptation methods in few-shot scenarios.

Summary by alphaXiv.