Reproduce “Towards Boosting LLMs-driven Relevance Modeling with Progressive Retrieved Behavior-augmented Prompting” on OpenResearch

About this paper

Zeyuan Chen, Haiyan Wu, Kaixin Wu, Wei Chen, Mingjie Zhong, Jia Xu, Zhongyi Liu, Wei Zhang

A new framework called ProRBP enhances large language models (LLMs) for industrial relevance modeling by integrating dynamic user behavior data with progressive prompting and an efficient online-offline deployment strategy. The system achieved an AUC of 0.9143 with GLM-10B on an Alipay dataset and demonstrated a 0.33% improvement in valid PV-CTR during live A/B testing, reducing irrelevant results by 1.07 percentage points without increased latency.

Key insights

• Effectively integrating dynamic, high-confidence user behavior data significantly boosts LLM performance in industrial relevance modeling by providing timely, domain-specific knowledge beyond general semantic understanding.
• Guiding LLMs with progressive, multi-aspect prompting improves their sensitivity and robustness to diverse relevance cues, leading to more accurate and stable judgments than isolated, naive prompting.
• A hybrid online-offline deployment strategy provides a practical solution for leveraging powerful LLMs in high-traffic industrial settings, achieving high accuracy with offline pre-computation while maintaining low latency with online distilled models.

Summary by alphaXiv.