Reproduce “Shared Actors Need Not Share Critics: Effects of Value Mismatch in Parallel Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhenya Liu, Yang Meng, Zhuokai Zhao, Xuefeng Liu, Yuxin Chen

The method trains a shared actor across environments, conditioning only its critic on a logged index. This lets the critic estimate environment-specific values without exposing the index to the actor, addressing shifts in sampled advantages from a shared baseline. On unseen Procgen levels, multihead reaches 140.8 ± 47.1%, versus 100.0 ± 19.8% for shared, aggregated across 16 games and 10 seeds.

Key insights

  • While different baseline choices converge to the same optimal policy asymptotically, their *realized learning paths* diverge significantly over finite horizons due to how baselines redistribute stochastic updates.
  • Value mismatch systematically shifts sampled advantages: promoting suboptimal actions in "easy" environments (where the shared value is too low) and suppressing optimal actions in "hard" environments (where the shared value is too high).
  • A conditional critic enables an eventual "ratchet" behavior, where every sample contributes to progress, contrasting with the "persistent drawdowns" caused by a shared critic due to unaddressed value mismatch.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.