Reproduce “Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Finance” on OpenResearch

About this paper

Fengbin Zhu, Junfeng Li, Liangming Pan, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, Tat-Seng Chua

Researchers from the National University of Singapore and collaborators at 6Estates Pte Ltd, Peking University, and the University of Science and Technology of China introduce FinTMMBench, a benchmark for temporal-aware multi-modal Retrieval-Augmented Generation (RAG) in finance, integrating tables, text, time-series, and visual data with time-sensitive questions. They also propose TMMHybridRAG, a method combining dense and graph retrieval with explicit temporal information, which achieves an F1 score of 31.41% on this new benchmark.

Key insights

• Retrieval-Augmented Generation is indispensable for accurately answering complex, knowledge-intensive financial questions, as evidenced by the poor performance of a 'No Retrieval' baseline (F1 5.89%).
• Explicitly incorporating temporal information into dense vectors and heterogeneous graph structures is crucial for effectively addressing time-sensitive queries and improving RAG performance in financial tasks.
• Converting diverse multi-modal data into unified textual descriptions for LLM processing, combined with retaining access to raw data during the generation phase, is an effective strategy for handling data heterogeneity and maintaining accuracy.

Summary by alphaXiv.