Reproduce “Mafin: Enhancing Black-Box Embeddings with Model Augmented Fine-Tuning” on OpenResearch

About this paper

Mingtian Zhang, Shawn Lan, Peter Hayes, David Barber

Mafin, a framework developed by researchers at UCL and Vectify AI, enhances black-box embedding models for domain-specific retrieval by augmenting them with a trainable white-box model. This method consistently improves retrieval accuracy in RAG systems, surpassing both fine-tuned open-source models and the original black-box models in supervised settings, and showing effectiveness even with unsupervised data generation.

Key insights

• Augmenting black-box embeddings with a smaller, trainable white-box model effectively adapts powerful proprietary models to specific domains without requiring access to their internal parameters.
• Combining diverse embedding representations, even if one is fixed, leads to more robust and specialized performance, a synergy observed even with simple concatenation.
• Large Language Models can be effectively leveraged to generate synthetic query-passage pairs, enabling domain-specific fine-tuning in data-scarce, unsupervised scenarios.

Summary by alphaXiv.