Reproduce “Russian-Language Multimodal Dataset for Automatic Summarization of Scientific Papers” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Alena Tsanda, Elena Bruches

Tsanda Alena and Bruches Elena developed the first publicly available multimodal dataset for Russian scientific paper summarization, featuring 420 papers across diverse domains with text, figures, and tables. Benchmarking against Russian large language models established baseline performance and highlighted the challenges in incorporating multimodal information for summarization.

Key insights

  • Humanitarian papers in the dataset tend to have longer texts with fewer figures and tables, while technical and natural science papers contain more visual information.
  • Current large language models (LLMs) used for benchmarking struggle with longer scientific texts due to context length limitations and do not effectively incorporate information from figures and tables.
  • Author-written 'Conclusion' sections from the papers consistently performed better than LLM-generated summaries across all evaluation metrics, highlighting the challenges in scientific paper summarization.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.