Reproduce “Interpreting Neural Networks through the Polytope Lens” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Sid Black, Lee Sharkey, Leo Grinsztajn, Eric Winsor, Dan Braun, Jacob Merizian, Kip Parker, Carlos Ramón Guevara and 3 more

Conjecture introduces the "polytope lens" as a framework for interpreting neural networks, asserting that ReLU-based models implement piecewise linear functions by partitioning input space into distinct polytope regions. This perspective reveals that neuron meanings are consistent within specific polytopes, explaining polysemanticity, and demonstrates that these geometric boundaries align with semantic shifts in both image and language processing.

Key insights

  • The meaning of individual neurons and feature directions in neural networks is often context-dependent, varying across different polytope regions created by non-linear activations.
  • Polytope boundaries in activation space often align with semantic boundaries in the input space, especially in higher layers of a trained network.
  • The network's local linear behavior within a polytope region defines when feature-directions are considered 'on-distribution', beyond which their interpretation changes or becomes meaningless.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.