Reproduce “Source-Aware Training Enables Knowledge Attribution in Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, Hao Peng

Researchers from the University of Michigan and the Allen Institute for AI developed a 'source-aware training' method that enables large language models to attribute their generated text to specific pretraining documents. This approach, which involves injecting unique document identifiers during pretraining and instruction tuning, achieved over 80% out-of-domain answer accuracy and up to 50% accurate source attribution (Hits@1) on a synthetic dataset, paving the way for more transparent and verifiable AI.

Key insights

  • Repeated document ID injection (REPEAT strategy) or Chain-of-Thought (CoT) attribution during instruction tuning are effective for enabling out-of-domain source attribution.
  • Document-level data augmentation (shuffling sentences within documents) is crucial for improving the model's generalization capability in out-of-domain attribution.
  • Source-aware training, particularly with the REPEAT and DOC-END + CoT setups, can slightly improve factual recall (answer Exact Match) compared to standard training.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.