Reproduce “Interpreting the Second-Order Effects of Neurons in CLIP” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yossi Gandelsman, Alexei A. Efros, Jacob Steinhardt
UC Berkeley researchers introduced a "second-order lens" to interpret individual neuron functions within CLIP's image encoder, analyzing their contributions through subsequent attention heads. This method enabled automated textual descriptions of polysemantic neurons and led to improved zero-shot image segmentation and the generation of semantic adversarial examples.
Key insights
- Neurons with the most significant contributions are concentrated in the moderately late MLP layers (8-10) of CLIP's image encoder.
- Individual neuron effects are highly sparse, contributing significantly for a small subset of images, and their second-order effect can be accurately approximated by a single direction in the output space.
- The automated textual descriptions confirmed polysemantic behavior, revealing that individual neurons often correspond to multiple, seemingly unrelated, semantic concepts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.