Reproduce “Tight PAC-Bayes Generalisation Guarantees for Large Language Model Judges” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tom A. Lamb, Philip H. S. Torr, Tim G. J. Rudner
The authors introduce LoRA-GT, an adaptation for certifying the risk of safety judges on prompt–response pairs from a fixed generator. It shares a quantization scale across layers, reducing description length, which enters a PAC-Bayes bound alongside observed risk. Across five models and four benchmarks, it has the tightest certificates in all 40 cases; guarantees do not cover deployment shifts.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.