Reproduce “Sanitizing Synthetic Training Data Generation for Question Answering over Knowledge Graphs” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Trond Linjordet, Krisztian Balog
Researchers at the University of Stavanger demonstrated that random partitioning of template-generated synthetic data for Knowledge Graph Question Answering (KGQA) leads to inflated performance due to information leakage, proposing a 'sanitized' partitioning scheme to accurately evaluate model generalization. Their method revealed a substantial performance drop (e.g., BLEU from 84.67% to 54.55% for NSpM+Att2) when models were tested on genuinely unseen templates.
Key insights
- Random partitioning of template-generated synthetic data significantly inflates perceived model performance due to information leakage, as models perform much worse on instances from genuinely unseen templates.
- Neural KGQA models can generalize from instances based on 'seen' templates to those from 'unseen' templates, and this generalization capability improves with a larger volume of training data.
- Increasing the proportion of unique templates available during training, even if only through more instances of those templates, can further enhance a model's ability to generalize to new templates, particularly reflected in perplexity scores.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.