Reproduce “EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, Pang Wei Koh

EvalTree organizes benchmark questions into a hierarchy of increasingly specific capabilities, using language-model descriptions, embeddings, clustering, and generated labels to identify low-performing nodes. On MATH, GPT-4o mini scores 75.1% on combinations and arrangements but 49.1% on geometric relationships using trigonometry. Synthetic MATH F1 is 0.7538; DS-1000 accuracy rises from 29.20% to 36.90% in tested setups.

Key insights

  • A hierarchical capability tree allows for simultaneous high-level interpretability and fine-grained diagnosis of LM performance.
  • Quantitatively assessing weakness profiling methods requires a suite of evaluations, including low-performance identification, ground-truth alignment, and extrinsic utility (e.g., in guiding training data collection).
  • Human-voter-based evaluations can exhibit biases or misalignments with desired model values, which structured profiling can help expose.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.