Reproduce “AuthorityBench: Benchmarking LLM Authority Perception for Reliable Retrieval-Augmented Generation” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhihui Yao, Hengran Zhang, Keping Bi
Researchers at the Chinese Academy of Sciences developed AuthorityBench, a benchmark to evaluate large language models' ability to perceive information authority from sources like web domains and entities. The study found that LLMs, particularly larger models, demonstrate this perception, and leveraging authority signals in Retrieval-Augmented Generation (RAG) tasks improves answer accuracy, outperforming other filtering strategies.
Key insights
- LLMs possess a discernible ability to perceive source and entity authority, with performance generally increasing with model scale.
- ListJudge and PairJudge methods, particularly when providing absolute scores (PointScore output), are most effective for eliciting robust authority judgments from LLMs.
- Incorporating webpage text can sometimes degrade domain authority judgments by confounding LLMs with surface features, but can also improve performance on hard comparisons.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.