Reproduce “Instance-wise Supervision-level Optimization in Active Learning” on OpenResearch

About this paper

Shinnosuke Matsuo, Riku Togashi, Ryoma Bise, Seiichi Uchida, Masahiro Nomura

An Instance-wise Supervision-level Optimization (ISO) framework is introduced to intelligently select data points for annotation and determine their optimal supervision level (full or weak). This approach consistently outperforms conventional active learning and state-of-the-art hybrid methods on CIFAR-100 and CUB-200, achieving comparable performance on CIFAR-100 in three rounds that baselines required five rounds to reach.

Key insights

• Not all unlabeled instances require expensive full supervision; some can contribute effectively with cheaper, coarser weak labels.
• Optimizing supervision decisions at the individual instance level is more effective than pre-determining fixed proportions for entire batches.
• The Value-to-Cost Ratio (VCR) framework effectively balances instance informativeness, expected model improvement, and annotation costs, while diversity in selected batches prevents redundancy.

Summary by alphaXiv.