Reproduce “GovRelBench:A Benchmark for Government Domain Relevance” on OpenResearch

About this paper

Haiquan Wang, Yi Chen, Shang Zeng, Yun Bian, Zhe Cui

Researchers from the Chinese Academy of Sciences introduce GovRelBench, a benchmark and evaluation tool designed to assess Large Language Models' relevance to the government domain. The benchmark features a specialized evaluation model, GovRelBERT, which employs a novel SoftGovScore method to quantify domain relevance as a continuous score, demonstrating superior performance and efficiency compared to larger general-purpose LLMs and traditional classification approaches.

Key insights

• Quantifying domain relevance as a continuous value via Beta diffusion (SoftGovScore) effectively models the inherent fuzziness and overlap of specialized domains, offering a more nuanced evaluation than strict classification.
• A specialized, smaller encoder-based model (GovRelBERT) can significantly outperform much larger general-purpose LLMs in domain-specific relevance assessment, demonstrating greater efficiency and focused capability.
• Direct regression to continuous relevance scores is more effective for fine-grained domain assessment than combined classification and regression approaches, as validated by ablation studies and expert judgment.

Summary by alphaXiv.