Reproduce “Instruction-tuned Language Models are Better Knowledge Learners” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhengbao Jiang, Zhiqing Sun, Weijia Shi, Pedro Rodriguez, Chunting Zhou, Graham Neubig, Xi Victoria Lin, Wen-tau Yih and 1 more

The authors introduce pre-instruction-tuning (PIT), which trains a model on question-answer examples before training it on those examples alongside documents; PIT++ adds an explicit question-only warm-up. They evaluate closed-book factual question answering on Wikipedia-derived Wiki2023-film questions. On Llama-2 7B, PIT++ reaches 48.1% exact match, versus 30.3% for standard instruction-tuning, within this test setup.

Key insights

  • LLMs suffer from a 'perplexity curse,' where achieving low perplexity on new documents through continued pre-training does not guarantee effective factual knowledge absorption or retrieval.
  • Prioritizing the learning of 'how knowledge is accessed' (through question-answer pairs) before or concurrently with 'encoding knowledge' from dense documents significantly enhances an LLM's ability to integrate new factual information.
  • The specific sequencing and mixing of instruction-tuning with document pre-training in the Pre-instruction-tuning (PIT) approach is crucial for optimizing knowledge learning, outperforming conventional sequential methods.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.