Reproduce “RocketEval: Efficient Automated LLM Evaluation via Grading Checklist” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, Jianghong Ma
Researchers from Tencent Youtu Lab and partner institutions introduce RocketEval, a breakthrough automated LLM evaluation framework that achieves comparable accuracy to GPT-4 while reducing costs by over 50x through innovative use of instance-specific checklists and lightweight judge models, demonstrating 0.965 Spearman correlation with human preferences on MT-BENCH.
Key insights
- Breaking down evaluation into specific checklist items enables lightweight models to match GPT-4's judgment accuracy
- Independent assessment of checklist items reduces interference and improves correlation with human preferences
- Weighted supervision based on KL divergence helps handle non-ideal human annotation distributions
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.