Reproduce “BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ryuhei Miyazato, Ting-Ruen Wei, Xuyang Wu, Hsin-Tai Wu, Kei Harada
BookAsSumQA evaluates aspect-based book summaries by generating genre-related questions from a book’s knowledge graph and testing whether summaries answer them. The authors compare methods across 30 books using metrics. For books over 200,000 words, NaiveRAG scores 20.55 ROUGE-1 versus 20.50 for GPT-4o-mini with Hierarchical Merging, while GPT + Hier is the overall top method; these scores are not validated against human judgments.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.