Reproduce “A Study of Position Bias in Digital Library Recommender Systems” on OpenResearch

About this paper

Andrew Collins, Dominika Tkaczyk, Akiko Aizawa, Joeran Beel

A large-scale study on over 10 million recommendations in digital library recommender systems empirically confirms the existence of strong position bias, showing that higher-ranked items receive 53% to 87% more clicks than expected in randomly shuffled lists, while also demonstrating that users actively identify and click relevant items regardless of their display rank.

Key insights

• Position bias is a pervasive and strong factor in digital library recommender systems, causing users to preferentially click items appearing at higher ranks even when their relevance is randomized.
• Despite strong position bias, a significant number of users actively scan and identify relevant items across all displayed ranks, particularly if the top-ranked recommendations are not highly relevant.
• The findings help reconcile inconsistencies between search engine research (showing strong bias) and some prior general recommender system studies (showing weaker or no bias) by demonstrating that both position bias and user discernment for relevance coexist in real-world academic contexts.

Summary by alphaXiv.