Reproduce “On The Specialization of Neural Modules” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Devon Jarvis, Richard Klein, Benjamin Rosman, Andrew M. Saxe

The paper analyzes whether modular architectures learn reusable compositional mappings when tasks provide case-specific cues. In a dataset family, its deep linear-network analysis finds compositional and non-compositional mappings remain coupled under dense and output-partitioned architectures; fully partitioning inputs and outputs separates them. For three-bit features, full-rank sampling probabilities are 57.5% with three examples and 91.9% with four, not accuracy rates.

Key insights

  • Systematicity requires the network's mapping between lower-rank (compositional) sub-structures to be orthogonal to other parts of its overall mapping.
  • Implicit biases from learning dynamics in dense networks are insufficient to promote module specialization; dense architectures inherently learn coupled mappings that prevent systematicity.
  • Only stringent, perfectly segregated architectural modularity, where modules are exclusively exposed to their designated sub-structures, enables effective systematic generalization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.