Reproduce “Soft Prompt Decoding for Multilingual Dense Retrieval” on OpenResearch

About this paper

Zhiqi Huang, Hansi Zeng, Hamed Zamani, James Allan

Researchers from the University of Massachusetts Amherst developed KD-SPD, a multilingual dense retrieval model that combines knowledge distillation and a novel soft prompt decoding architecture. This approach achieved superior retrieval performance, reduced language bias, and strong zero-shot transferability across 15 languages by leveraging robust monolingual English retrieval knowledge and parallel sentence data.

Key insights

• Leveraging knowledge distillation from a robust monolingual English retrieval model effectively 'translates' document representations from various languages into a common, high-performing embedding space.
• The novel decomposable soft prompt design allows for efficient sharing of universal linguistic knowledge while enabling language-specific adaptation, which is crucial for handling diverse languages within a single model.
• Utilizing abundant bitext data for distillation circumvents the need for expensive and scarce multilingual *retrieval* training labels, making the approach highly scalable and practical for low-resource languages.

Summary by alphaXiv.