Reproduce “Deeply-Supervised Nets” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, Zhuowen Tu

Deeply-Supervised Nets (DSN) train companion classifiers at hidden layers alongside the final classifier, sending label-based feedback into intermediate features during joint learning. On MNIST, DSN-SVM and CNN-SVM both have 0.03% training error but test error is 0.39% and 0.50%, respectively. This supports a generalization difference in this comparison, not a universal guarantee.

Key insights

  • Direct supervision at intermediate layers encourages the learning of more discriminative and transparent features throughout the entire network.
  • Injecting direct gradients at multiple points effectively mitigates the vanishing/exploding gradient problem, enabling more stable and effective training of deeper architectures.
  • The combined objective function with auxiliary losses can significantly accelerate convergence and act as a novel form of regularization, improving generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.