Reproduce “SkillAggregation: Reference-free LLM-Dependent Aggregation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Guangzhi Sun, Anmol Kagrecha, Potsawee Manakul, Phil Woodland, Mark Gales
SkillAggregation offers a neural framework for combining judgments from multiple large language model (LLM) judges without requiring ground truth labels. This approach dynamically weights each LLM's contribution based on contextual information and learned skill estimates, improving the reliability of LLM-based evaluations for tasks like hallucination detection and comparative assessment.
Key insights
- Effective aggregation of LLM judgments requires considering the specific evaluation context, rather than using static weights.
- Learning explicit skill estimates for each LLM judge enhances aggregation reliability, enabling the method to identify more dependable models for different evaluations.
- Regularization techniques are crucial for handling overconfident LLM predictions and improving the calibration of aggregated results.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.