Reproduce “A MISMATCHED Benchmark for Scientific Natural Language Inference” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Firoz Shaik, Mobashir Sadat, Nikita Gautam, Doina Caragea, Cornelia Caragea
The paper introduces MISMATCHED, a new benchmark dataset for scientific Natural Language Inference (NLI) designed to evaluate model generalization across diverse scientific disciplines beyond computer science. Evaluations reveal that fine-tuned small language models like SciBERT demonstrate superior out-of-domain performance compared to larger, prompt-based language models, with the 'REASONING' relation posing the greatest challenge.
Key insights
- Fine-tuned Small Language Models (SLMs) like SciBERT, when trained on existing scientific NLI data, achieved superior out-of-domain generalization compared to prompt-based Large Language Models (LLMs), including proprietary ones.
- The 'REASONING' relation type, which involves complex causal and inferential connections, posed the most significant challenge for all evaluated models.
- Incorporating 'implicit' NLI relations (adjacent sentences without explicit linking phrases) into training data measurably improved model performance, suggesting a valuable, underutilized signal.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.