Reproduce “Consistent Estimators for Learning to Defer to an Expert” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hussein Mozannar, David Sontag
The authors jointly train an image classifier and routing decision, treating deferral as an additional action. Their convex surrogate upper-bounds multiclass system error and is consistent at its population minimizer. On CIFAR-10, models improve average combined-system test accuracy over LearnedOracle by 1.94 percentage points across simulated-expert settings. These results do not establish performance with actual human experts.
Key insights
- The proposed $L_{CE}$ loss provides a theoretically sound, convex, and consistent surrogate for the non-convex system loss, ensuring that optimizing it leads to the Bayes optimal solution for deferral.
- Methods that independently train a classifier and an expert agreement model, then defer based on confidence scores, limit the classifier's ability to adapt to the expert's specific weaknesses, leading to suboptimal performance under constrained hypothesis spaces or limited data.
- Fairness metrics in human-AI systems are not necessarily compositional; independently fair machine learning components can contribute to an increase in overall system discrimination when combined with human expertise.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.