Reproduce “Last Translation Benchmark” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vilém Zouhar, Niyati Bafna, Mukund Choudhary, Maike Züfle, Sara Rajaee, Pinzhen Chen, Jannis Vamvas, Sara Papi and 252 more
The Last Translation Benchmark tests difficult, human-authored translation challenges with rules that check whether meaning is preserved. Its evaluation subset contains 911 text-only examples, selected for difficulty rather than typical use. On that subset, the LLM translator group’s verifier pass rate rises from 7.2% without rules to 89.8% with human-authored rules, showing performance when given targeted cues, not everyday quality.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.