Reproduce “Beyond Prompts: A Systematic Study of LLM-Based Machine Translation Evaluators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dmitry Popov, Roman Bokhyan, Ekaterina Enikeeva, Artem Mekhraliev, Stepan Vysotsky, Nikolay Karpachev
The authors study how to design LLM-based machine-translation evaluators that locate translation errors, testing instructions, examples, model backbones, and multi-step workflows. Their ORBIT-MC system fuses reports from three models. On English–Russian data, it reaches span F1 52.1 and major-error recall 82.0, above expert-agreement baselines, but MQM ranking remains lower; results depend on span matching and tested languages.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.