Reproduce “jp-evalb: Robust Alignment-based PARSEVAL Measures” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jungyeul Park, Junrui Wang, Eunkyul Leah Jo, Angela Yoonseo Park
jp-evalb introduces a robust evaluation system for constituency parsers that leverages alignment-based preprocessing to handle tokenization and sentence boundary discrepancies. This approach allows for accurate PARSEVAL measure computation in end-to-end NLP systems, successfully addressing known limitations of the traditional evalb script and providing a legacy mode for exact result replication.
Key insights
- Leveraging established alignment techniques from machine translation for intralingual evaluation can overcome rigid input requirements in parsing assessment.
- An alignment-based preprocessing step can make traditional evaluation metrics like PARSEVAL robust to real-world input variations without fundamentally altering the core measurement logic.
- Preserving compatibility with a long-standing evaluation standard (`evalb`) while enhancing robustness ensures that new research remains comparable with historical benchmarks.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.