Reproduce “From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoders” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yifan Luo, Yang Zhan, Jiedong Jiang, Tianyang Liu, Mingrui Wu, Zhennan Zhou, Bin Dong
Researchers from Peking University and Westlake University developed the Hierarchical Sparse Autoencoder (HSAE) framework, which jointly learns sparse features and their multi-scale conceptual hierarchies within Large Language Model (LLM) representations. This approach constructs semantically meaningful parent-child feature relationships, outperforming baseline SAEs in hierarchical consistency metrics and maintaining or improving feature quality while significantly reducing feature absorption.
Key insights
- Feature splitting, typically seen as a defect in SAEs, can be reinterpreted and actively exploited as direct evidence of an underlying hierarchical conceptual structure within LLMs.
- LLMs inherently organize complex concepts into fine-grained, semantically meaningful hierarchies that align with human intuition, which HSAE can successfully uncover.
- The learned semantic hierarchies are intrinsically represented in the geometric distribution of the LLM's activation space, with activations triggering sibling features clustering more closely than those triggering unrelated features.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.