Reproduce “Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization” on OpenResearch

About this paper

João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong

Researchers introduced a framework that leverages Direct Preference Optimization (DPO) to align synthetically generated web queries with ranking objectives, resulting in neural retrieval models that achieve superior performance on standard Web benchmarks. This method moves beyond post-hoc filtering by directly guiding the query generator to produce higher-quality training data.

Key insights

• Directly optimizing the query generator with DPO using ranking feedback is more effective and efficient than post-hoc filtering for improving synthetic data quality.
• Both specialized re-rankers and general-purpose LLMs can serve as effective reward mechanisms for DPO alignment, demonstrating the framework's flexibility.
• Aligning the generative model with a downstream task objective significantly enhances the utility of synthetic data for training dense retrieval models, enabling state-of-the-art performance.

Summary by alphaXiv.