Reproduce “Personalization for Web-based Services using Offline Reinforcement Learning” on OpenResearch

About this paper

Pavlos Athanasios Apostolopoulos, Zehui Wang, Hanson Wang, Chad Zhou, Kittipat Virochsiri, Norm Zhou, Igor L. Markov

Researchers from The University of New Mexico and Facebook Inc. deployed Offline Reinforcement Learning to personalize user authentication in a major social network. The approach reduced One-Time Password (OTP) costs by 70.3% compared to a baseline policy and improved Return on Investment (ROI) by 5.93% over a Supervised Learning model, while maintaining user engagement.

Key insights

• Balanced exploration during data collection is crucial for mitigating distributional shift in Offline RL, leading to superior state coverage and higher cumulative rewards for trained models.
• Robust offline evaluation metrics, particularly Counterfactual Policy Evaluation, are essential for safely tuning and ranking RL models prior to costly online A/B testing and deployment.
• Offline RL can effectively balance conflicting long-term objectives like user engagement and operational costs, demonstrating an improved Return on Investment over traditional supervised learning approaches.

Summary by alphaXiv.