Reproduce “SIEVE: Sample-Efficient Parametric Learning from Natural Language” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Parth Asawa, Alexandros G. Dimakis, Matei Zaharia
Researchers at UC Berkeley developed SIEVE, a method for sample-efficient parametric learning that internalizes natural language context into large language models' weights using as few as three seed query examples. This approach, leveraging a novel synthetic data generation pipeline, matched or exceeded in-context learning performance across diverse reasoning and long-context tasks without requiring context during inference.
Key insights
- Natural language context is frequently decomposable, meaning only a subset of context units is applicable to any given query, which can be leveraged to create higher-quality training data.
- Selectively identifying and using only the applicable context units during synthetic data generation and teacher rollouts significantly improves the effectiveness and sample efficiency of context distillation.
- Parametric learning can achieve sample efficiency comparable to in-context learning by internalizing context directly into model weights, enabling persistent improvements for multi-step reasoning tasks without requiring context during inference.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.