Reproduce “EvoRank: LLM-Guided Evolution of Multi-Objective Learning-to-Rank Pipelines” on OpenResearch

About this paper

Rayhan Patel, Shabaz Patel

EvoRank uses an LLM-guided evolutionary search to modify learning-to-rank code, spanning ranking losses, feature construction, model choice, and ensembles. On Expedia hotel-search data, loss-only evolution mostly selected validation noise, while broader pipeline search produced held-out gains of 0.0057 in relevance NDCG@10 and 0.0056 in booking NDCG, with no statistically reliable revenue improvement.

Key insights

• Search-space headroom is decisive: loss-only changes were generally smaller than the selection fold’s noise floor, whereas manually constructed feature changes exceeded that floor and enabled transferable discovery.
• Increasing the fitness-set size reduced evaluation noise and improved transfer, while seeded knowledge and richer diagnostics mainly affected search speed or selection-fold progress rather than final generalization.
• The strongest recurring pipeline used interpretable within-query percentile and count features together with a small heterogeneous ensemble; leakage-prone hotel-level target encodings substantially harmed performance.

Summary by alphaXiv.