Reproduce “KuaiRec: A Fully-observed Dataset and Insights for Evaluating Recommender Systems” on OpenResearch

About this paper

Chongming Gao, Shijun Li, Wenqiang Lei, Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao and 1 more

KuaiRec is the first large-scale, real-world dataset providing nearly fully-observed user-item interactions from a short-video platform, designed to enable rigorous study of how data density and exposure biases impact recommender system evaluation. Experiments on KuaiRec demonstrate that biases and density significantly alter model performance and rankings, with standard matrix completion techniques offering only limited mitigation.

Key insights

• Exposure biases profoundly impact both the absolute performance and, more critically, the relative rankings of recommender system models, leading to untrustworthy conclusions from biased evaluation data.
• Even with uniformly random exposure, evaluating Conversational Recommender Systems in a realistic multi-target setting (MTG) shows significant instability in model performance and rankings with varying data density.
• Matrix completion methods only partially alleviate the issues of inconsistent rankings caused by biases or sparsity, especially for metrics like Average Turns, highlighting the fundamental challenge of bias propagation.

Summary by alphaXiv.