Reproduce “LM-CORE: Language Models with Contextually Relevant External Knowledge” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jivat Neet Kaur, Sumit Bhatia, Milan Aggarwal, Rachit Bansal, Balaji Krishnamurthy
LM-CORE, a framework developed by researchers at Microsoft Research and Adobe, enhances language models by explicitly providing contextually relevant external knowledge from structured knowledge bases. This approach significantly improves factual recall on the LAMA probe, achieving up to 42.83% Precision@1 compared to BERT-large's 25.44%, and enables dynamic knowledge updates without requiring model retraining.
Key insights
- Explicitly training language models to utilize external, contextually relevant knowledge significantly improves factual recall and reduces reliance on shallow linguistic cues compared to implicit knowledge storage.
- Decoupling the language model from the external knowledge source allows for dynamic knowledge updates without requiring computationally intensive model retraining.
- Structured knowledge bases, when effectively integrated via a retrieval mechanism, offer a more efficient and interpretable way to ground factual information in language models than embedding all knowledge into parameters.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.