Reproduce “Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling” on OpenResearch

About this paper

Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li

Researchers from the University of Science and Technology of China developed a Response-conditioned Modeling framework to explicitly disentangle length bias in preference learning and enhance Large Language Models' adherence to explicit length instructions. This approach, utilizing a Response-conditioned Bradley-Terry model, demonstrated improved semantic quality and precise length control for generated text across various models and datasets.

Key insights

• Explicitly disentangling length information from semantic quality in preference learning allows LLMs to develop a controllable understanding of response length, moving beyond mere suppression of length as a confounder.
• Augmenting preference data with conditions that specify length violations or satisfactions is essential for training models that can balance both semantic quality and precise length instruction following.
• The proposed framework is generalizable across different foundational models, diverse datasets (including out-of-distribution scenarios), and effective in mitigating other types of superficial biases beyond just length.

Summary by alphaXiv.