Reproduce “Optimizing Agent Behavior over Long Time Scales by Transporting Value” on OpenResearch

About this paper

Chia-Chun Hung, Timothy Lillicrap, Josh Abramson, Yan Wu, Mehdi Mirza, Federico Carnevale, Arun Ahuja, Greg Wayne

Temporal Value Transport (TVT) augments recurrent reinforcement learning with reconstructive external memory, using later memory retrievals to send value signals back to earlier events that contributed to delayed outcomes. In DeepMind Lab tasks with distractor intervals lasting up to 60 seconds and variable unrelated rewards, TVT enabled information gathering and causal credit assignment that standard recurrent and memory-based baselines often failed to learn, while reducing early-phase return variance by nearly two orders of magnitude in one key-to-door setting.

Key insights

• Memory storage alone was insufficient: reconstruction objectives helped preserve observations, actions, rewards, and value-relevant information for later retrieval.
• Later retrieval can serve as a credit-assignment signal, allowing the agent to bypass noisy distractor intervals rather than propagate reward information through every intervening step.
• The same mechanism can support both remembering information for a later decision and assigning value to earlier information-acquisition behavior, including multiple delayed dependencies.

Summary by alphaXiv.