Reproduce “Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover
Guidance Contrastive Policy Optimization (GCPO) introduces a mechanism for fine-grained, per-token credit assignment in discrete policy optimization by leveraging contrastive guidance derived from comparing model predictions under positive and negative prompts. The method improves text-to-image generation on GenEval by 0.09 points and consistently outperforms baselines in multimodal reasoning benchmarks.
Key insights
- Classifier-Free Guidance (CFG)-style contrast, traditionally used in image generation, can serve as a robust indicator of a token's semantic importance by quantifying its divergence between conditional and unconditional predictions.
- Applying a rank-based normalization to raw KL divergence values provides stable and consistent token importance weights, robustly handling varying scales across different sequences.
- Careful design of negative prompts, such as adding a 'generate a wrong answer' suffix for language tasks, allows the contrastive guidance mechanism to be effectively extended to multimodal reasoning scenarios without standard CFG.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.