Reproduce “Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto
Researchers at Stanford University developed Length-Controlled AlpacaEval (AlpacaEval-LC), a debiased automatic evaluator for large language models that uses a regression-based approach to mitigate length bias. This improved metric reduces susceptibility to output length manipulation from 25% to 10% normalized standard deviation in win rates and achieves a 0.98 Spearman correlation with human judgments from the LMSYS Chatbot Arena.
Key insights
- A simple, regression-based method effectively mitigates known biases in LLM evaluators, leading to a more accurate assessment of model performance.
- Controlling for length bias shifts leaderboard rankings, with proprietary models often improving due to their conciseness, while some verbose open-source models decrease.
- The proposed debiasing framework is generalizable, offering a blueprint for addressing other types of spurious correlations in automated evaluation metrics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.