Reproduce “Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tobias Schimanski, Jingwei Ni, Mathias Kraus, Elliott Ash, Markus Leippold
This research from the University of Zürich, ETH Zürich, and University of Regensburg demonstrates that high-quality synthetic data, generated with automated filtering, can significantly improve the faithfulness and robustness of open-source Large Language Models for evidence-based question answering. Their fine-tuned models achieved performance comparable to or exceeding closed-source counterparts like GPT-3.5 and GPT-4 on various tasks, and generalize well to diverse real-world datasets.
Key insights
- High-quality training data is more crucial than data quantity for fine-tuning LLMs in evidence-based question answering, with filtered synthetic data achieving substantial improvements.
- Fine-tuning with the proposed synthetic data positively transfers performance to diverse real-world and out-of-distribution scenarios, enhancing robustness for RAG applications.
- In-domain synthetic test sets can reliably predict out-of-distribution performance, showing a strong positive correlation (0.91-0.99 Pearson correlation) with real-world evaluations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.