Reproduce “InFoBench: Evaluating Instruction Following Ability in Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu and 2 more
INFOBENCH introduces a granular evaluation framework and the Decomposed Requirements Following Ratio (DRFR) metric to systematically assess Large Language Model instruction following by breaking down instructions into specific requirements. The study found that GPT-4 can reliably act as an evaluator, showing 93.5% agreement with human experts, and identified that LLMs struggle most with numerical constraints.
Key insights
- The Decomposed Requirements Following Ratio (DRFR) metric provides more granular and reliable evaluation of instruction following compared to traditional direct scoring.
- GPT-4 can serve as a highly accurate and cost-effective evaluator for LLM responses, achieving 93.5% agreement with human experts and significantly reducing evaluation time and expense.
- LLMs consistently perform best on 'Content' related instructions but struggle most with 'Number' constraints, indicating a weakness in precise numerical adherence.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.