Reproduce “The Lifecycle of LLM-as-a-Judge for Large-Scale Recommendation Explanations” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Emma Yanyang Kong, JJ Tan, Ishan Gupta, Lars Olds, Claire Campbell, David Fagnan, Ratna Kavuri, Veli Balin and 3 more
The paper presents a lifecycle for maintaining an LLM judge of recommendation explanations. It combines a benchmark, reasoning-aligned rubric tuning, production gating and revision, and recurring human monitoring. A five-week mobile A/B test against no explanations associated the pipeline with relative lifts of 0.2% toward unwatched titles and 0.3% in successful-play browsing sessions; it does not isolate the judge or RART.
Key insights
- Treating LLM judges as dynamic components with a full lifecycle, rather than static artifacts, is crucial for their long-term effectiveness in continuously evolving production systems.
- Aligning LLM judges with human rationales, not just final labels, significantly improves their specificity and ability to provide accurate feedback for generative model self-correction.
- A robust LLM-as-a-Judge system can measurably improve online user experience metrics, such as novel content discovery and reduced browse friction, while maintaining high explanation quality at scale.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.