Reproduce “Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Xing Li, Qingcheng Chang, Jinzhong Ning, Changfeng Xu, Shenlong Zhang, Yijia Zhang, Ling Luo, Hongfei Lin

Jev directly selects among supplied answers and returns probabilities over the options. The authors evaluate it on 13 multiple-choice benchmarks against 19 general-purpose language models across three tiers. It scores 95.0% on MMLU-Redux and 97.6% on ARC-Challenge, highest among evaluated models, but reaches 75.0% on MathQA, 17.7 points below the frontier median; this comparison does not establish the cause.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.