Reproduce “Less is More for Improving Automatic Evaluation of Factual Consistency” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tong Wang, Ninad Kulkarni, Yanjun Qi
LIM-RA is an advanced model for the automatic evaluation of factual consistency, achieving a 4.4% relative improvement in average AUC-ROC over previous state-of-the-art methods on a diverse set of benchmarks, including a new LLM-specific benchmark. It leverages a meticulously curated, smaller training dataset and demonstrates enhanced robustness to specific entity perturbations.
Key insights
- Optimizing training data through rigorous cleaning and strategic reduction can significantly enhance model performance, demonstrating that a smaller, higher-quality dataset is more effective than a larger, noisier one.
- Targeted augmentation with synthetic data, specifically designed to introduce subtle entity (name, number) perturbations, effectively boosts the model's robustness to critical factual inconsistencies.
- Selecting a more capable foundational model like DeBERTa-large, pre-trained on NLI tasks, provides a stronger base for fine-tuning factual consistency evaluators.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.