Reproduce “ACCEPT: Adaptive Codebook for Composite and Efficient Prompt Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yu-Chen Lin, Wei-Hua Li, Jun-Cheng Chen, Chu-Song Chen
Researchers from National Taiwan University and Academia Sinica introduce ACCEPT, an adaptive codebook approach for prompt tuning that subdivides prompt embeddings and uses shared learnable codebooks with adaptive weights. This method reduces trainable parameters to just 0.3% of the pre-trained language model, while improving average performance by 3.0% on GLUE and 4.4% on SuperGLUE compared to full fine-tuning.
Key insights
- Decomposing prompts into sub-components and leveraging shared, learnable codebooks significantly reduces trainable parameters while enabling more flexible and nuanced representations.
- Using soft-weighted combinations of codewords, rather than hard quantization, maintains differentiability and enhances the diversity and expressiveness of prompt learning.
- The approach effectively decouples the parameter count for prompt generation from the prompt length, achieving high efficiency, especially for longer prompts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.