Reproduce “Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings” on OpenResearch

About this paper

Liyan Xu, Zhenlin Su, Mo Yu, Jiangnan Li, Fandong Meng, Jie Zhou

Researchers revealed that current dense retrievers struggle with fine-grained semantic matching, even on simple queries, leading to the identification of a "granularity dilemma" in embedding design. They introduced CapRetrieval, a human-annotated dataset for evaluating this issue, and demonstrated that fine-tuning with LLM-generated keywords can enhance fine-grained retrieval, achieving a 13% improvement on a 0.1B model compared to its original performance and surpassing larger models.

Key insights

• All evaluated state-of-the-art dense encoders, regardless of size or training source, consistently struggle with fine-grained query matching on the CapRetrieval dataset, demonstrating universal limitations in granular semantic understanding.
• A "granularity dilemma" exists in embedding training: while LLM-generated keywords improve fine-grained matching, they can inadvertently cause a model to lose its grasp on overall semantics or misalign semantic importance on other general retrieval tasks.
• Detailed error analysis revealed common failure modes for dense retrievers, including direct semantic misses, lack of taxonomy knowledge, absence of commonsense reasoning, and issues with over-generalization or ignoring critical query conditions.

Summary by alphaXiv.