Reproduce “Codebook Features: Sparse and Discrete Interpretability for Neural Networks” on OpenResearch

About this paper

Alex Tamkin, Mohammad Taufeeque, Noah D. Goodman

This research introduces "codebook features" as a method to make neural networks more interpretable and controllable by enforcing sparse, discrete internal representations. The approach involves finetuning models with vector quantization bottlenecks, demonstrating disentangled feature representation and causal steering of model behavior in both algorithmic and natural language tasks with only modest performance degradation.

Key insights

• Neural networks can operate effectively with only modest performance degradation even under extreme sparsity and discreteness imposed by codebook features, challenging assumptions about dense representations.
• Codebook features effectively mitigate the "superposition problem" by providing highly pure, disentangled representations of individual latent concepts, outperforming individual neurons.
• The discrete, on-or-off nature of codebook features provides an intuitive and direct interface for both understanding internal model function and causally steering model behavior.

Summary by alphaXiv.