Reproduce “CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Vipul Gupta, Zihao Wang, Razvan-Gabriel Dumitru, MohammadHossein Rezaei, Aakash Sabharwal, Yunzhong He
CRAFT introduces a methodology for diagnosing specific weaknesses in large language models by clustering rubric-annotated evaluation data into a hierarchical tree of capabilities. This framework then guides the generation of targeted fine-tuning data, leading to improved model performance, with fine-tuned models achieving up to 8.4 points higher average accuracy in finance and 4.1 points higher in legal domains compared to baseline data selection methods.
Key insights
- Rubric criteria serve as more precise and informative diagnostic units for LLM evaluation compared to whole prompts or task-level scores, enabling a finer-grained understanding of model failures.
- Organizing capabilities into a semantic hierarchy with dynamic selection of weak nodes allows for identifying and addressing model weaknesses at the most appropriate conceptual depth.
- Directly linking fine-grained evaluation outputs (diagnosed weak capabilities) to targeted data generation establishes a more efficient and effective feedback loop for continuous LLM improvement.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.