Reproduce “Semantic Residual Prompts for Continual Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Martin Menabue, Emanuele Frascaroli, Matteo Boschini, Enver Sangineto, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara
STAR-Prompt is a continual learning method that addresses catastrophic forgetting in prompt-tuning by ensuring stable prompt selection and effective task adaptation. It achieves a 5.96% average improvement in Final Average Accuracy over state-of-the-art methods across diverse datasets and demonstrates resilience to significant domain shifts.
Key insights
- Catastrophic forgetting in prompt-tuning methods can occur within the prompt selection mechanism itself, an overlooked problem where prompt keys become unstable over time.
- Decoupling the requirements for stability (using a foundation model like CLIP for prompt key generation) and plasticity (adapting a separate backbone like ViT) through a two-level prompting strategy is highly effective for continual learning.
- Injecting task-specific information as 'semantic residuals' directly into the ViT's residual connections, after attention and before the MLP, ensures better integration of prompt knowledge compared to conventional prefix tuning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.