Reproduce “LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, Krishna Kamath
Researchers at Pinterest developed and deployed a system utilizing fine-tuned Large Language Models for automated relevance assessment in web-scale search A/B experiments. This approach significantly enhanced the efficiency and sensitivity of relevance metrics, including a substantial reduction in Minimum Detectable Effects, by enabling larger and more sophisticated sampling designs.
Key insights
- Fine-tuned LLMs demonstrate strong alignment with human relevance judgments, exhibiting high agreement rates and negligible average bias, especially for paired experiment differences.
- The operational efficiency of LLM labeling enables the adoption of stratified sampling, which is crucial for reducing variance and improving the sensitivity of relevance metrics.
- Multilingual LLMs effectively extend automated relevance assessment to non-English markets, offering cost reductions and efficiency gains across a global user base.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.