Reproduce “Prediction-Powered LLM Ranking with Adaptively Collected Human Feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Gabriel Sargent, Will Wei Sun, Zhengwu Zhang, Yufeng Liu
PP-Rank combines LLM-judge predictions on unrated prompts with human comparisons to estimate human-preference rankings under adaptive model-pair selection. It corrects judge contributions with human feedback and inverse-probability weights. In semi-synthetic trials with simulated preferences, it certified the top model in 0.74 of trials versus 0.52 for labeled-only inverse-propensity weighting; this was not a live human evaluation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.