Reproduce “Reinforcement Learning with Token-level Feedback for Controllable Text Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Wendi Li, Wei Wei, Kaihe Xu, Wenfeng Xie, Dangyang Chen, Yu Cheng
A reinforcement learning framework, TOLE, introduces token-level feedback for controllable text generation, which enhances control accuracy, maintains text quality, and allows for efficient multi-attribute steering. This approach leverages the marginal contribution of each token to attribute fulfillment, leading to superior performance in sentiment control, detoxification, and multi-constraint generation tasks.
Key insights
- Fine-grained (token-level) feedback provides more precise guidance for LLMs in reinforcement learning, leading to faster convergence and better control than coarse-grained feedback.
- Quantizing and injecting noise into rewards can prevent reward hacking and maintain text diversity in reinforcement learning-based text generation, making the training more robust.
- A learnable weighting mechanism can effectively resolve contradictions when controlling multiple text attributes simultaneously, enabling flexible and efficient multi-attribute generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.