Reproduce “sk-bench: A Native-First Benchmark for Evaluating Large Language Models in Slovak” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Marek Šuppa, Ivan Vykopal, Andrej Ridzik, Kristián Sopkovič, Natália Kňažeková, Jaroslav Kopčan, Miroslav Blšták, Viktória Ondrejová and 4 more

sk-bench evaluates open and proprietary models on Slovak tasks using mostly native data and selected translated proxies in a shared generate-and-match pipeline. Across 55 models, native exams and translated multiple-choice rankings correlate at ρ = 0.98, while human-authored and generated Slovak question-answering rankings correlate at ρ = 0.72. These dataset-level comparisons cover standard written Slovak and text-only, single-turn tasks.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.