Reproduce “zELO: ELO-inspired Training Method for Rerankers and Embedding Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Nicholas Pipitone, Ghita Houir Alami, Advaith Avadhanam, Anton Kaminskyi, Ashley Khoo

ZeroEntropy AI introduces zELO, an unsupervised training method for rerankers and embedding models that leverages LLM ensembles to generate high-quality pairwise preferences. This ELO-inspired approach enables the `zerank` models to achieve state-of-the-art retrieval performance, surpassing commercial rerankers and LLM-as-rerankers across various domains.

Key insights

  • The 'Laffer curve' effect in hard negative mining indicates that overly intelligent negative miners can degrade model performance by selecting negatives more relevant than positives.
  • Large Language Model (LLM) ensembles can generate higher-quality training data than human annotators, enabling scalable and cost-effective unsupervised data generation.
  • ELO-inspired statistical models, like the Thurstone model, can effectively convert pairwise preferences into robust, absolute pointwise relevance scores, providing a theoretically grounded training signal.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.