Reproduce “Improving alignment of dialogue agents via targeted human judgements” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger and 26 more
Sparrow trains dialogue agents using human judgments of response preferences and compliance with specific natural-language rules, alongside adversarial probing data. At inference, it can search for evidence and rerank candidate answers. In human probes, it broke a targeted rule in 8% of cases; evidence-backed factual responses were rated plausible and supported 78% of the time, not verified as true.
Key insights
- Targeted human judgments on specific natural language rules provide more granular and effective feedback than broad safety labels, improving the mitigation of specific failure modes.
- A multi-objective RLHF approach successfully balances user preference with adherence to safety rules, demonstrating joint optimization of utility and safety.
- Integrating inline evidence from web searches significantly enhances the factual correctness and verifiability of dialogue agent responses, fostering greater user trust.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.