Reproduce “RewardRank: Optimizing True Learning-to-Rank Utility” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gaurav Bhatt, Kiran Koshy Thekumparampil, Tanmay Gangwani, Tesi Xiao, Leonid Sigal

RewardRank presents a two-stage framework that directly optimizes for true counterfactual utility in learning-to-rank tasks, moving beyond traditional proxy metrics. It leverages a learned, permutation-aware reward model and differentiable soft sorting, achieving higher counterfactual utility on novel reproducible benchmark suites and state-of-the-art relevance on real-world click data.

Key insights

  • Directly optimizing a data-driven, permutation-aware reward model for true counterfactual utility can more effectively bridge the offline-online gap in ranking systems compared to optimizing traditional proxy metrics.
  • Differentiable sorting operators, like SoftSort, provide a crucial mechanism for enabling end-to-end gradient-based optimization of rankers with respect to complex, non-differentiable rank-aware utility functions.
  • Large Language Models (LLMs) can function as robust and scalable "simulated users" or "oracles" for evaluating complex user behaviors and biases in counterfactual ranking scenarios, offering a reproducible alternative to A/B tests.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.