Reproduce “Evaluating Open-QA Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu and 2 more
This research introduces QA-Eval, a meta-evaluation task, and the EVOUNA dataset, the first human-annotated benchmark for assessing Open Question Answering (Open-QA) evaluators. The study demonstrates that current automatic evaluation metrics, including lexical matching, neural methods, and LLM-based evaluators, often misalign with human judgments, providing a framework for developing more reliable evaluation tools.
Key insights
- Existing automatic evaluators for Open-QA, including advanced LLM-based methods, generally fall short of human standards and often fail to rank Open-QA models consistently with human judgment.
- Specific error patterns vary by evaluator type: lexical methods struggle with semantic variations, neural methods with context, and LLM-evaluators exhibit sensitivity to answer length and literal interpretation.
- Prompt engineering techniques like Chain-of-Thought and In-Context Learning can improve LLM-based evaluator performance, particularly for longer answers, though their efficacy varies depending on the specific context and answer characteristics.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.