Reproduce “K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu ji, Guihong Cao, Daxin Jiang and 1 more
The K-ADAPTER framework introduces a modular method for infusing factual and linguistic knowledge into pre-trained language models like RoBERTa using compact, task-specific adapters. This approach consistently improves performance on knowledge-intensive downstream tasks while mitigating catastrophic forgetting.
Key insights
- Modular knowledge adapters enable continuous and incremental knowledge infusion into PLMs without altering the core model, thereby addressing catastrophic forgetting.
- The use of separate, independently trained adapters for distinct knowledge types leads to disentangled knowledge representations, offering better interpretability and control.
- Infusing specific factual and linguistic knowledge via compact adapters effectively enhances PLMs' performance on knowledge-intensive downstream tasks more efficiently than full model retraining or multi-task approaches.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.