Reproduce “Decoupled Entropy Minimization” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jing Ma, Hanlin Li, Xiang Xiang

Researchers at HUST AI and Visual Learning Lab reformulated classical Entropy Minimization, decoupling it into distinct components to reveal and address its inherent limitations. Their Adaptive Decoupled Entropy Minimization (AdaDEM) method, which is hyperparameter-free, consistently improves performance and robustness across diverse imperfectly supervised learning benchmarks, including test-time adaptation and semi-supervised learning.

Key insights

  • Classical Entropy Minimization's internal mechanics can be understood by decoupling it into CADF (drives cluster aggregation) and GMC (mitigates gradients, pushes to uniformity) with opposing effects.
  • The decoupled view accurately identifies "reward collapse" and "easy-class bias" as fundamental limitations that restrict EM's effectiveness in practice.
  • Adaptively normalizing reward signals and dynamically estimating marginal label entropy are crucial for overcoming EM's biases and improving its robustness without manual hyperparameter tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.