Reproduce “DualEval: Joint Model-Item Calibration for Unified LLM Evaluation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Aaron J. Li, Hao Huang, Youngmin Park, Yitong Ma, Wei-Lin Chiang, Li Chen, Cho-Jui Hsieh, Bin Yu and 1 more
DualEval is a framework that unifies static benchmark correctness labels and open-ended arena-style preferences for evaluating Large Language Models, jointly estimating model abilities and item characteristics using Item Response Theory. The framework improves ranking stability, identifies redundant evaluation items for cost reduction, and enables anomaly detection, allowing for more efficient and auditable assessment.
Key insights
- Jointly modeling model abilities and item characteristics (difficulty, sharpness) within a unified framework provides a more robust and balanced assessment than relying on single-source evaluations.
- Quantifying item informativeness via Fisher information allows for the identification of redundant items, enabling significant benchmark compression and cost reduction without sacrificing ranking fidelity.
- Per-pair residual analysis facilitates effective anomaly detection, providing a principled method to audit evaluation pipelines for potential data contamination or quality issues.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.