Reproduce “Transformer Key-Value Memories Are Nearly as Interpretable as Sparse Autoencoders” on OpenResearch

About this paper

Mengyu Ye, Jun Suzuki, Tatsuro Inaba, Tatsuki Kuribayashi

This research systematically evaluates the interpretability of features inherently present in Transformer Feed-Forward (FF) layers, termed FF-KV memories, comparing them against features discovered by external Sparse Autoencoders (SAEs) and Transcoders. The study demonstrates that FF-KV features offer comparable interpretability across various benchmarks, often achieving advantages in explained variance and concept absorption, without requiring additional training or parameters.

Key insights

• Features organically represented in Transformer FF layers (FF-KVs) provide interpretability nearly equivalent to, and in some aspects superior to, features discovered by external Sparse Autoencoders (SAEs).
• Human annotators found it challenging to qualitatively distinguish the origin of features from FF-KVs, SAEs, and Transcoders based solely on activation patterns, indicating similar perceived interpretability.
• Significant misalignment was observed between FF-KV and Transcoder features, raising questions about whether proxy modules truly decompose existing features or potentially 'hallucinate' entirely new representations.

Summary by alphaXiv.