Reproduce “Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Ryan Swift, Konstantinos Psounis

Follow the Features (FtF) selects instruction-tuning examples by identifying recurring patterns in a language model’s internal activations with a sparse autoencoder, then prioritizing examples that express selected features. In Qwen 3 8B Base fine-tuned on Alpaca Cleaned, its 100-example subset scores 20.51 HMean versus 18.92 for full-data training, but GRAPE scores higher and full data leads on the leaderboard aggregate.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.