Reproduce “Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hongda Sun, Yuxuan Liu, Chengwei Wu, Haiyu Yan, Cheng Tai, Xin Gao, Shuo Shang, Rui Yan
LLMQA proposes a unified framework for open-domain question answering that leverages a single large language model to perform multiple distinct roles—Generator, Reranker, and Evaluator—to improve answer accuracy and evidence quality. The system integrates query expansion, document reranking, and an automatic prompt optimization method, yielding state-of-the-art performance on standard ODQA benchmarks like Natural Questions, WebQuestions, and TriviaQA.
Key insights
- Large Language Models (LLMs) can effectively perform multiple specialized tasks (Generator, Reranker, Evaluator) when guided by appropriate prompts within a unified framework.
- Automatic prompt optimization, utilizing a variational inference-based approach, is crucial for effectively harnessing LLMs in their various roles.
- The collaborative interaction and feedback loop between LLM roles, where one role's output is evaluated and refined for subsequent steps, enhances overall Open-Domain Question Answering (ODQA) performance.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.