Reproduce “DelTA: Discriminative Token Credit Assignment for Reinforcement Learning from Verifiable Rewards” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Kaiyi Zhang, Wei Wu, Yankai Lin
Researchers from Renmin University of China and Ant International introduced DELTA, a framework that enhances reinforcement learning from verifiable rewards (RLVR) in large language models by assigning token-level credit based on a novel "discriminator view" of policy updates. DELTA reweights token-gradient terms to amplify discriminative signals, consistently improving mathematical reasoning scores by up to 3.79 points on various LLM backbones and generalizing to code generation and out-of-domain tasks.
Key insights
- RLVR policy gradient updates define an implicit linear discriminator that determines which token probabilities are increased or decreased based on their token-gradient vectors.
- Standard RLVR's simple advantage-weighted averaging for side-wise centroids can dilute discriminative signals, as common task-agnostic patterns appear in both good and bad responses.
- By adaptively reweighting token-gradient terms based on their discriminative signal, the effective side-wise centroids can be made more contrastive, leading to more targeted policy updates.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.