Reproduce “Pessimistic Off-Policy Multi-Objective Optimization” on OpenResearch

About this paper

Shima Alizadeh, Aniruddha Bhargava, Karthick Gopalswamy, Lalit Jain, Branislav Kveton, Ge Liu

AWS AI Labs researchers developed Pessimistic Off-Policy Multi-Objective Optimization (pessHVI), an a-posteriori framework that robustly discovers a diverse set of Pareto-optimal policies from logged data. This method, which optimizes a lower confidence bound on policy values, consistently achieves superior recovered hypervolume compared to existing baselines across multi-objective contextual bandit benchmarks.

Key insights

• Pessimistic optimization of the hypervolume's lower confidence bound offers stronger theoretical robustness guarantees, as its performance error depends solely on the uncertainty of the true optimal policy set, unlike mean-estimate optimization.
• The benefit of pessimistic optimization is most pronounced when the logging policy is non-uniform, effectively mitigating the higher variance and uncertainty introduced by poor coverage for certain target policies.
• The paper quantifies the hypervolume confidence interval width, showing dependencies on reward noise, the number of optimized policies, the number of objectives, and the sample size.

Summary by alphaXiv.