Reproduce “MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Song Feng, Siva Sankalp Patel, Hui Wan, Sachindra Joshi

IBM Research AI introduces MultiDoc2Dial, a new dataset and task for modeling goal-oriented dialogues that are dynamically grounded in multiple documents. Their baseline retriever-reader models demonstrate the importance of fine-tuned dense retrievers and structural document segmentation, achieving 43.5 F1 for grounding span prediction, while also highlighting challenges in domain generalization and handling conversational ambiguities.

Key insights

  • Fine-tuning dense retrievers on the specific multi-document dialogue data is crucial for achieving effective retrieval and generation performance.
  • Utilizing document structural information (e.g., HTML markup) for passage segmentation generally leads to improved retrieval recall and overall model performance.
  • Generating coherent natural language responses (Task II) is substantially more challenging for models than predicting specific grounding spans (Task I), despite comparable retrieval performance.
  • Generalizing multi-document grounded dialogue models to entirely new, unseen domains presents a significant performance drop, indicating a need for better domain adaptation strategies.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.