Reproduce “Long-Context Generation Is a Sampling Problem (When You Turn Up the Heat)” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Allen G. Roush, Minh Nhat Nguyen, Ravid Shwartz-Ziv, Judah Goldfeder, Sanjay Basu
The paper tests sampling rules for high-temperature long-form generation and coding agents. Top-nσ keeps tokens within a chosen number of standard deviations of the highest logit, making its candidate set invariant to temperature for fixed logits. At temperature 2, it resolves 51 of 100 SWE-bench Verified issues on Qwen3.5-9B, versus zero for top-p; this result is limited to that model and issue set.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.