Reproduce “Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Gleb Gerasimov, Timofei Rusalev, Nikita Balagansky, Daniil Laptev, Vadim Kurochkin, Daniil Gavrilov

Researchers at T-Tech analyzed seed dependence in sparse autoencoders (SAEs), demonstrating that individual features often vary across training runs but collectively form reproducible low-rank subspaces. They found stable features are functionally more impactful for reconstruction and prediction, while unstable features typically correspond to lower-level lexical patterns with marginal impact.

Key insights

  • Individual feature instability in SAEs often arises from basis ambiguity within consistently learned, reproducible low-rank subspaces, rather than from complete absence of the underlying concept.
  • Stable features are functionally distinct and carry the majority of the reconstruction- and prediction-relevant signal, while unstable features often capture less impactful, lower-level lexical patterns.
  • Automatic interpretability metrics alone are insufficient for assessing SAE faithfulness, as SAEs trained on random models can yield plausible interpretations despite lacking feature reproducibility.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.