Reproduce “Domain-specific Question Answering with Hybrid Search” on OpenResearch

About this paper

Dewang Sultania, Zhaoyu Lu, Twisha Naik, Franck Dernoncourt, David Seunghyun Yoon, Sanat Sharma, Trung Bui, Ashok Gupta and 6 more

Adobe Inc. researchers developed a production-ready, generalizable framework for domain-specific question answering, utilizing a hybrid retrieval mechanism that linearly combines fine-tuned dense embeddings, BM25 keyword search, and host-based boosting. This approach achieved a Context nDCG of 0.847 and an LLM-based answer similarity of 0.780, outperforming single-retriever systems while demonstrating high robustness against inappropriate queries.

Key insights

• A hybrid retrieval strategy, integrating fine-tuned dense embeddings, sparse keyword search, and host-based authority boosting, demonstrably improves context relevance and answer quality in domain-specific QA.
• Empirical tuning of parameters like bm25 boost and host boost is crucial for optimizing the linear combination of retrieval signals, achieving a Context nDCG of 0.847.
• LLM-based evaluation metrics for groundedness and answer similarity show a strong correlation (0.89 Pearson coefficient) with human judgments, providing a scalable method for assessing QA system performance.

Summary by alphaXiv.