Reproduce “From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models” on OpenResearch

About this paper

Chenchen Zhang

The survey defines credit assignment as deciding which token, reasoning step, interaction turn, tool call, memory operation, or agent receives credit or blame for an outcome. It frames each claim by its unit, reference action, state, and downstream protocol, and gives a hidden-state example with effects of +1/2 and −1/2. Its audit covers 69 papers and a fixed 42-paper subset, but does not rank methods.

Key insights

• The evolution from reasoning to agentic LLM RL fundamentally reshapes credit assignment, driven by vastly longer trajectories, environment interactions, partial observability, heterogeneous actions, and non-verifiable intermediate states.
• Episode-level credit, while sufficient for early RLHF, becomes increasingly inadequate for agentic RL due to a collapse in signal-to-noise ratio over ultra-long horizons, leading to issues like the 'echo trap'.
• Without specific state restoration or valid observational assumptions, causal credit claims can be non-identifiable from text-only histories, emphasizing the need for rigorous evaluation controls and explicit declarations to ensure scientific validity.

Summary by alphaXiv.