Reproduce “LLMDistill4Ads: Using Cross-Encoders to Distill from LLM Signals for Advertiser Keyphrase Recommendations at eBay” on OpenResearch

About this paper

Soumik Dey, Benjamin Braun, Naveen Ravipati, Hansi Wu, Binbin Li

eBay Inc. researchers introduce LLMDistill4Ads, a multi-task knowledge distillation framework that leverages Large Language Models (LLMs) and cross-encoders to generate debiased advertiser keyphrase recommendations. This system achieved a 51.26% increase in Gross Merchandise Volume and a 38.69% increase in Return on Advertising Spend in an A/B test.

Key insights

• A two-stage knowledge distillation process (LLM Cross-Encoder Bi-Encoder) with Pearson Correlation Loss effectively transfers nuanced relevance judgments, outperforming direct distillation and achieving high fidelity to the teacher's ranking.
• Integrating LLM-generated supervision with existing CTR data and the distillation process (LLM+CTR+KD) is crucial for maximizing both the diversity of keyphrase recommendations and their relevance, as measured by LLM approval and retrieval pass rates.
• General-purpose LLMs serve as effective and scalable proxies for human judgment in generating debiased training labels, proving more robust than fine-tuned LLMs when human judgment data is scarce or biased by modality.

Summary by alphaXiv.