Reproduce “Online Label-Free Gradient Matching for Continual Language Model Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Sajjad Pakdamansavoji, Mohammad Reza Rezaei, Ilija Subasic, Pedro Garzon, Radu Paradovschi, Zhao Chen
OLGA compresses changing text data into synthetic examples for model updates. It matches gradients to real examples, using a frozen model’s next-token predictions instead of class labels, and maintains fixed-size rehearsal memory. Across five classification benchmarks with phi-1.5, it averages 75.5% offline accuracy and 73.6% stream-AUC, but results vary by task and do not establish performance on arbitrary data.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.