Reproduce “Resurrecting the Salmon: Rethinking Mechanistic Interpretability with Domain-Specific Sparse Autoencoders” on OpenResearch

About this paper

Charles O'Neill, Mudith Jayasekara, Max Kirkby

Researchers at Parsed, London, UK, introduce domain-specific Sparse Autoencoders (SAEs) for mechanistic interpretability, demonstrating that training SAEs on medical text yields a 15-20% higher Fraction of Variance Explained and more clinically relevant features compared to broad-domain SAEs. This approach effectively reduces linearly predictable reconstruction error, or "dark matter," in residual activations.

Key insights

• Domain-specific training of SAEs significantly improves reconstruction fidelity and reduces the 'dark matter' effect by more effectively capturing relevant linear features within a targeted domain.
• Features learned by domain-specific SAEs are more semantically distinct and align with meaningful concepts (e.g., clinical terms), enhancing interpretability compared to generic features from broad-domain SAEs.
• The findings suggest that the 'more is better' scaling paradigm for general-purpose SAEs does not translate directly to effective mechanistic interpretability; targeted domain-specific training offers a more efficient path.

Summary by alphaXiv.