Reproduce “KS-LLM: Knowledge Selection of Large Language Models with Evidence Document for Question Answering” on OpenResearch

About this paper

Xinxin Zheng, Feihu Che, Jinyang Wu, Shuai Zhang, Shuai Nie, Kang Liu, Jianhua Tao

KS-LLM enhances large language model accuracy in knowledge-intensive question answering by intelligently selecting and combining structured triples derived from the query with concise, relevant textual evidence from documents. The method consistently outperforms existing retrieval-augmented generation baselines across multiple QA datasets, demonstrating that precise, multi-modal knowledge input improves factual grounding.

Key insights

• Combining structured knowledge (triples) and concise textual evidence (selected sentences) provides a synergistic benefit, enhancing LLM knowledge absorption and answer accuracy.
• Intelligently selecting a small, optimal number of highly relevant sentences from evidence documents significantly improves performance by reducing information overload and noise, contrasting with the degradation seen when feeding full or overly long documents.
• The approach of deriving structured knowledge from the question itself, then using it to guide textual evidence selection, creates a focused and complementary context for LLMs.

Summary by alphaXiv.