Reproduce “EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, Pang Wei Koh
EvalTree organizes benchmark questions into a hierarchy of increasingly specific capabilities, using language-model descriptions, embeddings, clustering, and generated labels to identify low-performing nodes. On MATH, GPT-4o mini scores 75.1% on combinations and arrangements but 49.1% on geometric relationships using trigonometry. Synthetic MATH F1 is 0.7538; DS-1000 accuracy rises from 29.20% to 36.90% in tested setups.
Key insights
- A hierarchical capability tree allows for simultaneous high-level interpretability and fine-grained diagnosis of LM performance.
- Quantitatively assessing weakness profiling methods requires a suite of evaluations, including low-performance identification, ground-truth alignment, and extrinsic utility (e.g., in guiding training data collection).
- Human-voter-based evaluations can exhibit biases or misalignments with desired model values, which structured profiling can help expose.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.