Reproduce “Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning” on OpenResearch

About this paper

Zongmeng Zhang, Yufeng Shi, Jinhua Zhu, Wengang Zhou, Xiang Qi, Peng Zhang, Houqiang Li

This paper presents TRUSTWORTHY-ALIGNMENT, a reinforcement learning method that teaches retrieval-augmented Large Language Models to prioritize external retrieved information over their internal knowledge, even when the two conflict. The approach improves model trustworthiness by consistently grounding responses in provided contexts and demonstrates reduced performance degradation on general tasks compared to supervised fine-tuning.

Key insights

• LLMs can be effectively aligned to prioritize external retrieved context over their internal parametric knowledge, even when conflicting, by framing it as a reinforcement learning task.
• A carefully designed reward function incorporating context alignment, KL divergence regularization, and a collapse penalty is crucial for achieving trustworthy behavior without degrading other model capabilities.
• Unlike supervised fine-tuning, RL-based alignment for trustworthiness can preserve the model's original language style and significantly reduce the 'alignment tax' on general tasks.

Summary by alphaXiv.