Reproduce “FashionSAP: Symbols and Attributes Prompt for Fine-grained Fashion Vision-Language Pre-training” on OpenResearch

About this paper

Yunpeng Han, Lisai Zhang, Qingcai Chen, Zhijian Chen, Zhonghua Li, Jianxin Yang, Zhao Cao

FashionSAP, a vision-language pre-training model, incorporates abstract fashion symbols and explicit attribute prompts to learn fine-grained multi-modal fashion characteristics. This approach achieved state-of-the-art performance across various fashion tasks, including cross-modal retrieval, category recognition, and text-modified image retrieval on FashionGen and FashionIQ datasets, demonstrating significant improvements, such as increasing mean R@1 from 63.71 to 71.63 on a FashionGen subset.

Key insights

• An abstract symbol layer can effectively standardize fashion item understanding and generalize fine-grained features across varied category schemes, improving implicit semantic capture from text to image.
• Explicitly leveraging prompt engineering with attribute key-value pairs during pre-training is critical for enhancing a VLP model's understanding of detailed visual and textual attributes in complex domains.
• Integrating domain-specific pre-training tasks that focus on fine-grained attributes significantly boosts performance in specialized VLP applications, as evidenced by the positive contributions of FSIS, PTP, and TRP.

Summary by alphaXiv.