Reproduce “GradientSpace: Unsupervised Data Clustering for Improved Instruction Tuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shrihari Sridharan, Deepak Ravikumar, Anand Raghunathan, Kaushik Roy
GradientSpace groups examples by LoRA update direction, trains an expert per group, and routes each incoming prompt to one expert without calculating its gradient at inference. A 500-pair MTEB STS sample showed weak positive input and gradient similarity (Pearson r = 0.249). On MATH with LLaMA-2-7B, it scored 46.8%, versus ELREA’s 41.8%; this system comparison does not isolate the cause.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.