Reproduce “Collective Dynamics of Self-Reinforcement: How Cross-Task Coupling Drives Policy Collapse in Unsupervised RLVR” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peize Ding

Authors analyze how unsupervised reinforcement learning with verifiable rewards can fail when models reward majority answers. Their model predicts shared directions accumulate signals across problems as problem-specific updates weaken with more problems. In a small Transformer, shared output bias caused collapse in 335 of 336 runs, versus none in 168 without bias; this result does not establish behavior in pretrained language models.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.