Reproduce “Optimal Use of Preferences in Artificial Intelligence Algorithms” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Joshua S. Gans

This paper establishes theoretical conditions for optimally integrating user preferences into AI algorithms, determining whether to embed them in training or apply them in post-prediction decision-making. It finds that separating preference-free prediction from ex post decision-making generally yields more informative signals and higher welfare, but embedding preferences can be optimal when decision-makers face significant cognitive costs.

Key insights

  • Embedding user preferences directly into an AI's training objective generally reduces the informativeness (dispersion) of the learned probability estimates.
  • Training with a preference-free, strictly proper scoring rule and applying preferences ex post through Bayes-optimal decision rules robustly dominates preference-embedded training for any expected-utility problem, preserving the option value of the predictor.
  • Decision-stage cognitive costs, such as those modeled by rational inattention, can reverse the separation principle, making preference embedding optimal by reducing the cognitive burden on decision-makers.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.