Reproduce “CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks” on OpenResearch

About this paper

Xiaoxi Li, Zhicheng Dou, Yujia Zhou, Fangchao Liu

CorpusLM introduces a unified language model framework that integrates generative retrieval, closed-book generation, and retrieval-augmented generation (RAG) into a single auto-regressive greedy decoding process for knowledge-intensive tasks. The model achieves superior retrieval performance, notably improving R-precision by up to +12.16% on T-REx compared to previous methods, and substantially enhances RAG performance while reducing memory footprint by 139.1-fold and query latency by 2.1-fold.

Key insights

• Unifying generative retrieval, closed-book generation, and RAG into a single, continuous auto-regressive decoding process with a shared language model significantly improves performance and efficiency for knowledge-intensive tasks.
• Learning to generate ranked lists of document identifiers (DocIDs) coupled with dynamic inference constraints leads to more robust and comprehensive retrieval compared to traditional generative retrieval methods.
• Explicitly decoding fine-grained references from retrieved documents before generating the final answer within a continuous sequence acts as an effective implicit filtering mechanism, reducing noise and improving RAG quality.
• Auxiliary unsupervised tasks designed to teach the LLM about DocID semantics enhance the model's internal representation of external knowledge, contributing to better retrieval and generation capabilities.

Summary by alphaXiv.