Reproduce “Efficient Dictionary Learning with Switch Sparse Autoencoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Anish Mudide, Joshua Engels, Eric J. Michaud, Max Tegmark, Christian Schroeder de Witt

Switch Sparse Autoencoders (Switch SAEs) introduce an architecture that integrates Mixture of Experts principles to efficiently extract interpretable features from large language models. This approach reduces the computational cost of training, achieving lower reconstruction error for a given FLOP budget, and enables scaling to a greater number of features than traditional Sparse Autoencoders.

Key insights

  • Conditional computation, typically used in Mixture of Experts (MoE) models, can be successfully adapted to Sparse Autoencoders, leading to enhanced computational efficiency in feature extraction.
  • Increasing the number of experts in Switch SAEs improves compute efficiency, but this often correlates with a reduction in parameter efficiency due to increased feature duplication.
  • Switch SAEs can achieve Pareto improvements in reconstruction error vs. sparsity when compute-matched against traditional SAEs, indicating efficient scaling of features at constant computational cost.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.