Reproduce “Mistral-SPLADE: LLMs for better Learned Sparse Retrieval” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Meet Doshi, Vishwajeet Kumar, Rudra Murthy, Vignesh P, Jaydeep Sen

Researchers from Indian Institute of Technology Bombay and IBM Research developed Echo-Mistral-SPLADE, an advanced Learned Sparse Retriever that adapts a decoder-only Mistral-7B LLM with Echo embeddings. This model achieved state-of-the-art zero-shot retrieval performance on the BEIR benchmark, averaging 55.07 nDCG@10 and surpassing previous SPLADE variants and some dense retrievers, while simplifying the training process.

Key insights

  • Large decoder-only LLMs, when appropriately adapted, are highly effective at learning robust semantic keyword expansions for sparse retrieval, surpassing the capabilities of smaller encoder-only models.
  • Architectural challenges of causal LLMs, such as unidirectional attention and untied weights, can be effectively addressed through techniques like Echo embeddings and LoRA finetuning without compromising the benefits of large pre-trained models.
  • State-of-the-art sparse retrieval performance can be achieved with a simplified training pipeline, relying on diverse datasets and in-batch negatives, potentially reducing the need for complex, resource-intensive strategies like hard negative mining.
  • An LLM-based sparse retriever can deliver competitive, and in certain zero-shot out-of-domain scenarios, even superior performance compared to similar-sized LLM-based dense retrieval models, while retaining interpretability and efficiency advantages.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.