Reproduce “Training Language Models to Generate Text with Citations via Fine-grained Rewards” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Chengyu Huang, Zeqiu Wu, Yushi Hu, Wenya Wang

Researchers from National University of Singapore, University of Washington, and Nanyang Technological University developed a training framework employing fine-grained rewards for reinforcement learning and rejection sampling, enabling large language models to generate text with verifiable in-text citations. This method allowed a LLaMA-2-7B model to achieve better attributable generation performance than ChatGPT, improving average evaluation metrics by up to 10.6% on the ELI5 dataset.

Key insights

  • Fine-grained reward signals, which provide specific feedback on correctness, citation recall, and citation precision, are more effective than holistic rewards for training attributable text generation.
  • Optimizing for a single objective (e.g., correctness only) can degrade performance on other crucial aspects (e.g., citation quality), emphasizing the necessity of a balanced multi-objective training approach.
  • Initial distillation from a high-performing model like ChatGPT significantly improves the baseline ability of smaller LLMs to generate in-text citations, serving as an effective starting point for further fine-tuning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.