Reproduce “PersonalAI: A Systematic Comparison of Knowledge Graph Storage and Retrieval Approaches for Personalized LLM agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Mikhail Menschikov, Dmitry Evseev, Victoria Dochkina, Ruslan Kostoev, Ilia Perepechkin, Petr Anokhin, Nikita Semenov, Evgeny Burnaev
PersonalAI introduces a flexible external memory framework for personalized LLM agents, utilizing a hybrid knowledge graph automatically constructed and updated by an LLM, paired with diverse retrieval mechanisms. The system achieved a 14.1% improvement over existing GraphRAG methods in specific configurations and demonstrated competitive performance on dialogue-based and multi-hop reasoning benchmarks.
Key insights
- Optimal memory configuration and retrieval strategy are task-dependent, with specific algorithm and traversal constraint combinations yielding better performance for different LLM scales and datasets.
- Thesis-type vertices frequently contain critical information for smaller LLMs, while larger models can be more robust to noisy episodic memories, impacting optimal traversal constraints.
- Hybrid retrieval strategies, combining algorithms like BeamSearch and WaterCircles, improve overall recall and stability across varying graph constraints, reducing the frequency of "NoAnswer" responses.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.