Reproduce “CARM: Cancellation-Aware Response Masking for LLM Reinforcement Learning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yafei Zhang, Songshuo Lu, Sicong Liao, Zhi Chen, Yaohua Tang
CARM adds a response-level gate to LLM reinforcement learning by averaging the absolute log changes in sampled-token probabilities and rejecting responses above a threshold. Unlike GeoMean’s signed average, this measure does not let opposing shifts cancel; it complements token-level clipping. On Qwen3.5-9B, CARM reaches 70.42% across four math benchmarks versus GeoMean’s 67.29%, with gains on all four for that model.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.