Reproduce “Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jonathan Cook, Chris Lu, Edward Hughes, Joel Z. Leibo, Jakob Foerster

Cook et al. from Oxford FLAIR and Google DeepMind demonstrated that emergent cultural accumulation can arise in reinforcement learning agents, leading to sustained generational performance gains across diverse tasks. Their models, which balance social learning and independent exploration, consistently outperformed single-lifetime agents with equivalent cumulative experience budgets.

Key insights

  • Emergent cultural accumulation is achievable in RL, demonstrating that agents can organically balance imitation and innovation across generations without explicit hand-crafted losses.
  • Cultural accumulation is sensitive to the accuracy of the observed prior generation; optimal learning requires a balance where the oracle is neither too accurate (leading to over-reliance) nor too unreliable.
  • In-weights accumulation effectively mitigates 'primacy bias' in deep RL, where agents converge prematurely based on early experiences, suggesting a robust training benefit.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.