Reproduce “KNN-LM Does Not Improve Open-ended Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shufan Wang, Yixiao Song, Andrew Drozdov, Aparna Garimella, Varun Manjunatha, Mohit Iyyer
Interpolation-based retrieval-augmented language models like kNN-LM and TRIME achieve substantial perplexity reductions but do not yield corresponding improvements in open-ended text generation quality. This arises from the retriever's limited token-level benefits and its declining reliability due to exposure bias during long sequence generation.
Key insights
- Lower perplexity in interpolation-based RALMs (kNN-LM, TRIME) does not guarantee better open-ended text generation quality.
- kNN-LM only improves the perplexity of a minority (42%) of individual gold tokens, potentially worsening predictions for the majority.
- The retriever's reliability deteriorates due to exposure bias as generated sequence length increases, leading to less confident and more divergent predictions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.