Reproduce “From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Eric Zhao, Pranjal Awasthi, Nika Haghtalab
Research from Google Research and UC Berkeley systematically explores the boundaries of finetuning for knowledge injection, demonstrating that the distinction between style adaptation and factual learning is artificial. This work identifies critical factors, such as training data format and information type, that determine finetuning success, showing question-answer pairs achieved up to 98% accuracy for certain facts while numerical data proved more challenging.
Key insights
- Training data format significantly impacts finetuning effectiveness, with question-answer (QA) pairs consistently outperforming formats like Wikipedia-style articles for knowledge injection.
- Categorical information is more readily integrated by models than numerical information, which poses a greater challenge.
- The quantity of information inversely affects finetuning accuracy, showing a power-law decline as the number of facts increases.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.