Reproduce “WebGPT: Browser-assisted question-answering with human feedback” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain and 10 more
OpenAI's WebGPT system integrates a large language model (GPT-3) with an interactive text-based web-browsing environment to generate long-form, factually accurate answers by actively searching and quoting information from the live web. The model's answers, with collected references, are preferred by humans over those from human demonstrators and highly-voted Reddit answers, while also demonstrating improved truthfulness on adversarial questions.
Key insights
- Endowing large language models with interactive web-browsing capabilities significantly improves their ability to dynamically acquire and synthesize up-to-date, factual information, reducing "hallucinations."
- Leveraging human feedback for alignment, specifically with human preference comparisons facilitated by collected web references, effectively trains models to generate high-quality, verifiable long-form answers.
- Rejection sampling at inference time, where multiple answers are generated and ranked by a reward model, can be a highly effective strategy for optimizing answer quality, often outperforming or complementing reinforcement learning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.