Reproduce “Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, Jitendra Malik

Side-tuning adapts pretrained models by keeping the base model fixed, training a task-specific side network, and combining their representations. In incremental learning, separate side networks preserve learned task paths without changing earlier ones. On iTaskonomy, side-tuning achieved an average rank of 1.33 versus Progressive Neural Networks’ 2.42, but on iCIFAR its rank was worse: 2.20 versus 1.10.

Key insights

  • By keeping the base model frozen and training independent side networks for each task, side-tuning inherently prevents catastrophic forgetting and rigidity in continual learning settings.
  • The approach effectively balances the bias-variance trade-off in transfer learning, leveraging the strong prior from the base model while allowing for task-specific adaptation through the side network.
  • Side-tuning is model-agnostic, treating the base model as a black box, which allows its application to a wide range of pre-trained models, including non-neural network sources of information.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.