Reproduce “BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg
BitFit, developed by researchers at Bar Ilan University and AI2, proposes a method for fine-tuning transformer-based masked language models by training only bias terms and the classification layer. This approach achieves performance comparable to full fine-tuning on GLUE benchmarks while modifying as little as 0.08% of total model parameters and often outperforming full fine-tuning in data-scarce scenarios.
Key insights
- Bias terms play a unique and critical role in adapting pre-trained Transformer models, as demonstrated by the superior performance of BitFit compared to fine-tuning other small, random parameter subsets.
- Fine-tuning primarily involves adapting or re-calibrating the extensive general linguistic knowledge already present in the pre-trained model, rather than learning entirely new task-specific features.
- The bias terms associated with query projections and the second MLP layer are particularly influential, allowing strong performance by modifying as little as 0.04% of total parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.