Reproduce “Sparse Fine-Tuning of Transformers for Generative Tasks” on OpenResearch

About this paper

Wei Chen, Jingxi Yu, Zichen Miao, Qiang Qiu

Researchers at Purdue University introduced a sparse fine-tuning framework for transformer models that represents adapted features as a sparse combination of dictionary atoms. This method demonstrated enhanced interpretability, controllability, and robustness in generative tasks such as image editing and text-to-image concept customization, outperforming existing dense fine-tuning baselines.

Key insights

• Tuning the 'feature dictionary atoms' is more effective for adapting to new tasks than tuning the sparse coefficients, as these atoms inherently encode the new task's representation.
• Complex generative concepts can be efficiently represented by a small set of essential dictionary atoms (e.g., four atoms for core structure, approximately 40 for a full concept).
• The discrete activation/inactivation nature of sparse coefficients provides greater stability and robustness against minor input perturbations compared to continuous updates in dense fine-tuning methods.

Summary by alphaXiv.