Reproduce “What pass@k Cannot Measure: Evaluating Diversity and Capability Retention after Post-Training” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Subham Rath, Raj Dandekar, Rajat Dandekar, Sreedath Panat
The authors show why pass@k measures correctness but cannot establish output diversity. They compare GRPO and RFT from one starting checkpoint after math post-training: both RFT arms increase three output-diversity measures, while GRPO decreases them. Yet held-out GSM8K pass@32 is similar: 0.969 for GRPO and 0.972 for RFT-shortest. This comparison covers one checkpoint and math tasks; it does not identify why the methods differ.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.