Reproduce “SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Amirhossein Abaskohi, Issam H. Laradji, Peter West, Giuseppe Carenini
SPROUTRAG is an attention-guided hierarchical RAG framework that builds a binary tree from sentence-level chunks using learned inter-sentence attention, enabling precise multi-granularity retrieval for long documents. It achieved superior Information Efficiency, improving by an average of 6.1% over baselines, while significantly reducing online computational costs compared to LLM-heavy systems.
Key insights
- Fine-tuned SLLM attention can implicitly capture robust structural information beyond textual proximity, providing a strong signal for building meaningful document hierarchies.
- Learned weighted aggregation of attention heads is crucial to overcome proximity bias in uniform attention averaging, enhancing the model's ability to identify relevant multi-sentence relationships.
- A multi-granular hierarchical structure, combined with efficient beam search retrieval, allows for a better balance between precision and contextual coverage, leading to more coherent and less noisy retrieved evidence without incurring high computational costs.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.