Reproduce “Towards Robust Universal Information Extraction: Benchmark, Evaluation, and Solution” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jizhao Zhu, Akang Shi, Zixuan Li, Long Bai, Xiaolong Jin, Jiafeng Guo, Xueqi Cheng

Researchers from ICT-CAS introduce RUIE-Bench, a comprehensive benchmark for evaluating Universal Information Extraction (UIE) robustness, along with Loss-guided Data Augmentation (LDA), an efficient training method that achieves 7.5% performance gains while using only 15% of augmented data through strategic sample selection based on inference loss.

Key insights

  • Existing UIE models, including LLM-based ones, show significant performance drops under perturbations
  • Open-source LLMs lag behind closed-source models in both standard and robustness evaluations
  • Strategic selection of training samples based on inference loss is more effective than using all augmented data

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.