Reproduce “Retrieval, Scoring, and Decoding Shape Performance and Stability in LLM-based Conversational Recommendation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ante Kapetanovic, Tomislav Duricic, Andro Mercep, Emanuel Lacic
Infobip researchers systematically evaluate how pipeline choices impact the performance and stability of LLM-based rerankers in conversational recommender systems. The study quantifies that factors like first-stage retriever, candidate-pool size, scoring policies, and decoding configurations critically influence measured ranking quality and list stability, with proprietary LLMs demonstrating substantial performance gains and varying stability depending on these choices.
Key insights
- Proprietary LLMs can achieve superior reranking performance, particularly in zero-shot or full-catalog scenarios, but their measured effectiveness is profoundly shaped by how candidates are retrieved and scored.
- The choice of the first-stage retriever and the size of the candidate pool significantly influence observed LLM reranker quality, often having a greater impact than intrinsic LLM differences.
- LLM mean ranking accuracy can remain stable across decoding temperatures, while list-level agreement (stability) substantially decreases, highlighting list stability as a critical, under-reported metric for real-world deployments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.