Reproduce “On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jingyi Cui, Qi Zhang, Yifei Wang, Yisen Wang

A theoretical framework for Sparse Autoencoders (SAEs) is provided, revealing their limitations in fully recovering ground truth monosemantic features due to phenomena like feature shrinking and vanishing, except under extreme sparsity conditions. The proposed Reweighted Sparse Autoencoders (WSAEs) mitigate these issues, achieving an average gain of 3.8% in auto-interpretability scores on Pythia-160M and improving semantic consistency in ResNet-18 vision features.

Key insights

  • Standard SAEs generally fail to fully recover ground truth monosemantic features, exhibiting phenomena like feature shrinking (recovered features are smaller) and feature vanishing (features disappear entirely) under typical sparsity conditions.
  • Full feature recovery by SAEs is theoretically guaranteed only when ground truth monosemantic features are "extremely sparse," providing a specific condition under which SAEs perform optimally.
  • A measurable "gap" exists between the standard SAE objective (reconstructing polysemantic inputs) and the ideal goal of reconstructing unobserved ground truth monosemantic features, which can be mitigated by reweighting the loss to assign smaller weights to more polysemantic dimensions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.