Reproduce “Network of Theseus (like the ship)” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Vighnesh Subramaniam, Colin Conwell, Boris Katz, Andrei Barbu, Brian Cheung

Network of Theseus (NoT) introduces a method to convert a neural network's architecture used for training into an entirely different one for inference by progressively replacing components and aligning their internal representations. This approach enables smaller, specialized models to achieve performance comparable to larger, more complex architectures, even when guided by untrained networks.

Key insights

  • Neural network architectures, independent of learned weights, contain valuable inductive biases that can be effectively transferred even from completely untrained guide networks.
  • Architectural priors primarily function as 'training biases' or scaffolds that aid and stabilize optimization, rather than immutable components essential for the deployed model's functionality.
  • The ability to decouple training and inference architectures allows for the exploration of a broader range of inference-time designs that may offer superior accuracy-efficiency trade-offs but are difficult to train from scratch.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.