Reproduce “Le Critique: Privileged Value Functions for LLM Reinforcement Learning” on OpenResearch

About this paper

Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

The paper proposes two ways to improve value-function baselines for language-model reinforcement learning: privileged value functions let critics use admissible training-only clues, while TETHER blends token-level values with a leave-one-out group baseline. On CodeIO, Qwen3-4B-Instruct-2507 at K=4 reached 0.542 with PVF versus 0.517 for VF and 0.518 for the group mean, with no measured speed or compute savings.

Key insights

• Providing value functions with privileged information during training significantly enhances their predictive accuracy and ability to reduce policy gradient variance, leading to improved learning efficiency without altering the policy optimum.
• An adaptive mixture of group-level and token-level baselines (TETHER) offers a robust strategy for integrating value functions, as it dynamically adjusts its reliance on the value function as its accuracy improves, minimizing risk.
• Value-based methods, particularly when enhanced by privileged information or adaptive blending, are highly sensitive to the GAE hyperparameter near 1.0, indicating that precise tuning can unlock substantial performance gains.

Summary by alphaXiv.