Reproduce “cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers” on OpenResearch

About this paper

Anirudh Sundar, Jin Xu, William Gay, Christopher Richardson, Larry Heck

The cPAPERS dataset provides 5,030 multimodal conversational question-answer pairs grounded in 2,350 scientific papers, derived from peer-review interactions from OpenReview and arXiv. This dataset enables the development of AI systems capable of engaging in meaningful discussions about scientific content by integrating text with equations, figures, and tables. Baseline experiments demonstrate that providing relevant multimodal context improves question-answering performance, with fine-tuned LLMs achieving competitive results.

Key insights

• Neighboring multimodal elements (specifically referenced or near the discussion) provide improved performance for answering questions compared to using all equations/tables/figures.
• The optimal context type for answering questions varies by modality: question-only performs well for equation-based QA, referring text is more effective for table-based QA, and a combination of question and relevant context works best for figure-based QA.
• Fine-tuning a smaller LLM (LLAMA-2-7B) on the dataset achieves competitive results, with the question alone proving effective for table and figure-based questions, suggesting reviewers integrate visual elements into their questions.

Summary by alphaXiv.