Reproduce “Multimodal Multi-Hop Question Answering Through a Conversation Between Tools and Efficiently Finetuned Large Language Models” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Hossein Rajabzadeh, Suyuchen Wang, Hyock Ju Kwon, Bang Liu
The authors present an LLM strategy that asks modality-specific tools simple questions, using each reply to choose the next. This breaks multi-hop tasks into smaller lookups. On MMCoQA, ChatGPT scores 47.05 EM and 58.82 F1 with 3.80 average tool calls, versus mm-ReAct’s 41.33, 52.17, and 6.35; this comparison does not isolate the cause of the gains.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.