Reproduce “BoRP: Bootstrapped Regression Probing for Scalable and Human-Aligned LLM Evaluation” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Peng Sun, Xiangyu Zhang, Duan Wu, Lu Tan, Jian Lin, He Yang, Qi Qian, Yikai Wang

Alibaba's Qwen Business Unit developed BoRP (Bootstrapped Regression Probing), a discriminative framework that extracts continuous user satisfaction scores directly from large language model hidden states. This approach achieved a Krippendorff’s alpha of 0.80, outperforming generative LLM-as-a-Judge methods, while simultaneously offering a 9.3x speedup and reducing inference costs by over 30 times.

Key insights

  • Directly extracting satisfaction signals from an LLM's latent space through probing (the 'read, don't write' paradigm) inherently mitigates common biases of generative evaluators and yields stronger alignment with human expert judgments.
  • Leveraging geometric signatures like Euclidean norm and Manhattan distance of difference vectors allows for effective polarization mining in unlabeled data, enabling the automated bootstrapping of evaluation rubrics from scratch.
  • An intrinsic uncertainty measure, derived from prediction differences between intermediate and final transformer layers, provides a zero-cost proxy for prediction reliability, enabling intelligent tiered evaluation and optimized resource allocation.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.