Reproduce “Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Laura Kopf, Nils Feldhus, Kirill Bykov, Philine Lou Bommer, Anna Hedström, Marina M.-C. Höhne, Oliver Eberle

Researchers from Technische Universität Berlin and BIFOLD developed PRISM, a framework to generate multi-concept textual descriptions for Large Language Model features, explicitly addressing polysemanticity. PRISM achieves superior description quality compared to prior methods and provides empirical evidence that individual LLM features encode diverse concepts, with its polysemanticity scores aligning with human judgments.

Key insights

  • Individual LLM features, including MLP neurons and sparse autoencoder features, frequently exhibit polysemanticity, encoding multiple distinct concepts rather than just one.
  • Providing multi-concept descriptions significantly enhances the accuracy and granularity of feature interpretations, leading to a more faithful understanding of LLM internals.
  • Interpretability and the degree of polysemanticity are highly dependent on the specific model and network layer, with middle layers often yielding more interpretable concepts.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.