Reproduce “Context-Action Embedding Learning for Off-Policy Evaluation in Contextual Bandits” on OpenResearch

About this paper

Kushagra Chandak, Vincent Liu, Haanvid Lee

CAEL-MIPS learns context-action embeddings to directly minimize the Mean Squared Error (MSE) of the Marginalized IPS (MIPS) estimator for off-policy evaluation in contextual bandits. This method consistently achieves the lowest MSE on both synthetic and real-world datasets, outperforming existing baselines by significantly reducing bias.

Key insights

• Directly optimizing the MIPS estimator's MSE through explicit bias and variance reduction terms in the loss function leads to superior performance compared to proxy objectives like reward prediction.
• Learning context-action dependent embeddings is crucial for capturing the true value of actions across diverse contexts, leading to more accurate and robust off-policy evaluation.
• The derived upper bounds for bias and variance provide a principled mechanism to manage the bias-variance trade-off within the embedding learning process, enhancing overall estimator precision.

Summary by alphaXiv.