Reproduce “Small Language Model Makes an Effective Long Text Extractor” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yelin Chen, Fanjin Zhang, Jie Tang
A lightweight span-based model, SeNER, was developed to perform Named Entity Recognition on long texts, which traditionally pose computational and memory challenges. The model achieves state-of-the-art F1 scores on long NER benchmarks while significantly improving GPU memory efficiency and inference speed compared to previous methods.
Key insights
- Balancing global contextual understanding (via `[CLS]` token attention) with local, fine-grained information (via sliding window attention) is crucial for effective and efficient long text encoding.
- Modeling token-pair span interactions efficiently by focusing on a relevant subset of candidate spans and using tailored attention mechanisms (BiSPA) drastically reduces memory and computation for long texts.
- Techniques like LogN-Scaling for attention stability and LoRA for efficient fine-tuning are vital for model robustness and practical deployment in resource-constrained long text scenarios.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.