Reproduce “AdapterDistillation: Non-Destructive Task Composition with Knowledge Distillation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junjie Wang, Yicheng Chen, Wangshu Zhang, Sen Hu, Teng Xu, Jing Zheng
AdapterDistillation introduces a two-stage knowledge distillation algorithm for integrating new natural language processing tasks into multi-tenant platforms, utilizing parameter-efficient adapters. This method enables non-destructive task composition, achieving performance comparable to AdapterFusion with substantially lower resource usage and faster inference speeds.
Key insights
- Knowledge from multiple existing task adapters can be effectively distilled into a new task-specific adapter, enabling composite learning without requiring a permanent fusion layer.
- Discarding the fusion mechanism after training allows the student adapter to encapsulate combined knowledge, maintaining inference speed and low parameter overhead.
- This two-stage distillation approach facilitates non-destructive integration of new tasks into multi-tenant systems, preserving the performance of previously deployed tasks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.