Reproduce “Dynamic Injection of Entity Knowledge into Dense Retrievers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ikuya Yamada, Ryokan Ri, Takeshi Kojima, Yusuke Iwasawa, Yutaka Matsuo
The Knowledgeable Passage Retriever (KPR) extends BERT-based dense retrievers by dynamically injecting external entity knowledge via a context-entity attention layer, enabling real-time knowledge updates without model retraining. This approach enhances retrieval accuracy, particularly for queries involving less-frequent entities, and demonstrates robustness to noisy entity linking.
Key insights
- Dynamic injection of external, pre-computed entity embeddings through a dedicated attention layer, while keeping the base retriever frozen, enables efficient knowledge updates without retraining.
- The context-entity attention mechanism is robust to noise and ambiguity from a simple entity linker, effectively selecting relevant entities based on context.
- Utilizing a sigmoid activation function with a length-based bias in the attention layer proves more effective than softmax for independently assessing entity relevance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.