Reproduce “Instance Selection Mechanisms for Human-in-the-Loop Systems in Few-Shot Learning” on OpenResearch

About this paper

Johannes Jakubik, Benedikt Blumenstiel, Michael Vössing, Patrick Hemmer

Researchers from Karlsruhe Institute of Technology systematically evaluated instance selection mechanisms for Human-in-the-Loop systems integrated with Prototypical Neural Networks for few-shot learning. Their evaluation on real-world datasets demonstrated that these systems substantially improve model accuracy with reduced human labeling effort, identifying confidence-based and newly adapted ensemble-based methods as most efficient.

Key insights

• Hybrid intelligence, through HITL systems, dramatically improves few-shot learning performance, showcasing up to a 48.4% increase in model accuracy on challenging datasets.
• Instance selection mechanisms, especially confidence-based (e.g., Minimum Confidence) and ensemble-based (e.g., BALD), are crucial for efficiency, outperforming random selection.
• Effective instance selection leads to substantial cost savings, requiring significantly less human labeling effort (e.g., 7.9% to 21.1% less) to achieve high model accuracy compared to random approaches.

Summary by alphaXiv.