Reproduce “Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorization” on OpenResearch

About this paper

Felipe Maia Polo, Aida Nematzadeh, Virginia Aglietti, Adam Fisch, Isabela Albuquerque

Researchers developed a tensor factorization-based statistical model that integrates extensive automated rater data with sparse human annotations to enable efficient, fine-grained evaluation of generative AI models. This approach accurately predicts human preferences and diagnoses model strengths and weaknesses at the prompt or category level, even when human labels are scarce.

Key insights

• Model performance is a composite of low-dimensional latent "skills" which can be efficiently learned via a low-rank tensor decomposition across models, prompts, and raters.
• A two-stage fitting process effectively transfers robust representations learned from abundant noisy autorater data to accurately predict human preferences using only a small calibration set of human labels.
• The statistical model provides tight and quantifiable confidence intervals, enabling trustworthy, fine-grained model comparisons and the diagnosis of specific strengths and weaknesses.

Summary by alphaXiv.