Reproduce “SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao Wang

SO-Arena evaluates one-pass System One models on decision capability and reliability. Capability averages task credit across domains, emphasizing difficult items; reliability separately combines consistency under option reordering with recognizing when the correct answer is absent. GLiDE ties KnowLine-31B at 70.2 capability but ranks seventh in reliability, while JevAny-27B leads reliability at 0.84. These are benchmark results, not deployment guarantees.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.