Reproduce “Seq@K: Evaluating Language Models Under Sequential Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Thomson Yen, Gilbert Yang, Naimeng Ye, Minghui Chen, Dong Xia, Tianyi Shen, Yechan Jeon, Yaqin Chen and 3 more
The authors introduce Seq@K, a measure of whether a model succeeds within K attempts when it receives feedback after failures. Unlike Pass@K, it evaluates revision across feedback-linked tries. On HumanEval Pro, selecting a model with Seq@3 led to 96.8% held-out Seq@5 success, versus 70.8% with Pass@5, at lower mean calibration API cost; this result is limited to the reported setup.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.