Reproduce “Misaligned by Design: Incentive Failures in Machine Learning” on OpenResearch

About this paper

David Autor, Andrew Caplin, Daniel Martin, Philip Marx

This paper reveals that directly integrating asymmetric error costs into a machine learning model's loss function, a common practice, consistently leads to suboptimal performance. It introduces an economically-grounded alternative where models are trained with an unweighted loss for better probability calibration, and human utility is applied through an ex-post adjustment mechanism, yielding superior results in high-stakes classification tasks.

Key insights

• Machine learning classifiers simultaneously perform two incentivized tasks: "choosing how to classify" based on learned information, and "learning how to classify" by acquiring and refining that information.
• While direct utility-weighted loss functions correctly incentivize the "choosing" task (e.g., biasing towards costly errors), they inadvertently dampen the incentives for the more fundamental "learning" task.
• Applying utility weights flattens the marginal value of acquiring more precise information across the probability space, thereby reducing the machine's motivation to form a more informative representation of the data.

Summary by alphaXiv.