Reproduce “Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Léo Hemamou, Mehdi Debiane
The EYEGLAXS framework from Sanofi leverages Large Language Models for extractive summarization of long documents by incorporating extended position encoding, Flash Attention 2, and parameter-efficient fine-tuning. This approach demonstrates competitive ROUGE scores compared to traditional encoder-only models on scientific datasets and shows strong performance even with limited training data.
Key insights
- LLMs can be effectively adapted for extractive summarization of long documents by combining techniques like position interpolation for extended context, Flash Attention 2 for efficiency, and LoRA for parameter-efficient fine-tuning.
- Models fine-tuned with longer context lengths (e.g., 12K tokens) consistently perform better for extractive summarization than those with shorter context (4K tokens).
- LLMs like LLAMA2-7B demonstrate high data efficiency, achieving strong performance even with limited training data, suggesting effective knowledge transfer from pre-training.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.