Reproduce “Refined Assessment for Translation Evaluation: Rethinking Machine Translation Evaluation in the Era of Human-Level Systems” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Dmitry Popov, Vladislav Negodin, Ekaterina Enikeeva, Iana Matrosova, Nikolay Karpachev, Max Ryabinin
The RATE protocol evaluates English-to-Russian translation by marking errors and assessing source-meaning accuracy separately from fluency. Professional linguists applied it to a selected WMT24 sample spanning four domains and compared a human reference with seven top-rated systems. Dubformer scored higher in accuracy, while the human reference scored higher in fluency; this split applies to the sampled translations, not translation generally.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.