Reproduce “MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages” on OpenResearch

About this paper

Chenxi Whitehouse, Sebastian Ruder, Tony Lin, Oksana Kurylo, Haruka Takagi, Janice Lam, Nicolò Busetto, Denise Diaz and 1 more

Meta Superintelligence Labs developed MENLO, a framework and dataset for evaluating and improving the native-like quality of Large Language Model (LLM) responses across 47 language varieties. The work demonstrates that reinforcement learning (RL) trained LLM judges can effectively guide policy models to enhance native-like proficiency, with human evaluation confirming improvements in output quality.

Key insights

• Pairwise evaluation consistently and significantly outperforms pointwise scoring for LLM judges in assessing native-like quality, with detailed rubrics further enhancing performance.
• Reinforcement learning (RL) with reward shaping proved more effective than SFT for training LLM judges, achieving higher agreement with human annotators and surpassing frontier API models in certain metrics.
• Localized Factuality remains a persistent challenge for LLM judges, showing limited benefits from RL training and suggesting a need for external tools or retrieval mechanisms.
• While RL-trained LLM judges can effectively improve policy model quality, they tend to overestimate the magnitude of these improvements compared to human evaluators, indicating a potential for stylistic "caricature" without careful calibration.

Summary by alphaXiv.