Reproduce “Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discovery” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Wang, Yifan Yang, TingHao YU, Difan Zou
The authors develop chunk-level sparse autoencoders that represent pooled language-model spans, with one variant predicting a neighboring span. Mean-Chunk yields high-level concepts in 34.8% of features versus 17.2% for BatchTopK, and reaches 66% cue-free backtracking recall versus under 5% for token-level baselines. Cross-Chunk reaches 77.0% same-document Recall@5 versus 64.1% for Temporal. These results concern tested tasks, not superiority across tasks or efficiency gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.