Reproduce “Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models” on OpenResearch

About this paper

Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, Dacheng Tao

A new prompting strategy, Error Analysis Prompting (EAPrompt), enhances large language models' capabilities in machine translation quality assessment by enabling detailed error identification and scoring. This method achieved a state-of-the-art 91.2% pairwise accuracy at the system level with GPT-3.5-Turbo and significantly improved segment-level evaluation, providing explainable error analysis aligned with human Multidimensional Quality Metrics (MQM).

Key insights

• Guiding LLMs with a structured error analysis process, mirroring human evaluation frameworks like MQM, substantially improves their performance in MT quality assessment at both system and segment levels.
• Specific prompt design choices, such as using itemized error examples and separating error identification from counting, are critical for enhancing LLM accuracy and the interpretability of their evaluations.
• LLMs, when appropriately prompted, can effectively distinguish between major and minor translation errors and generate error distributions that closely align with human annotations.

Summary by alphaXiv.