Reproduce “Topic-Conversation Relevance (TCR) Dataset and Benchmarks” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Yaran Fan, Jamie Pool, Senja Filipi, Ross Cutler
Microsoft researchers introduce the Topic-Conversation Relevance (TCR) dataset, the first specifically designed to measure how well conversations align with pre-defined meeting topics. Benchmarks utilizing GPT-4-32k demonstrate strong performance in highly structured meeting scenarios while highlighting challenges in less formal discussions.
Key insights
- Advanced LLMs like GPT-4 are highly effective at curating and transforming existing data (post-meeting summaries) into a format suitable for new, complex NLP tasks (pre-meeting agenda topic generation).
- Meeting structure profoundly influences the accuracy of topic-conversation relevance assessment; highly structured meetings are significantly easier to analyze than free-flowing, less formal discussions.
- The granularity and style of topic annotations (ground truth) heavily impact model performance, suggesting the importance of consistent and well-defined topic boundaries.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.