Reproduce “Personalize-then-Store: Benchmarking and Learning Personalized Memory for Long-horizon Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yeonjun In, Wonjoong Kim, Sangwu Park, Kanghoon Yoon, Chanyoung Park

The paper introduces PerMem-Bench and a session-level memory gate that predicts whether each session merits long-term storage, skipping memory operations for transient interactions. The benchmark models user-specific domains, projects, and reference memories across timelines. With Qwen3-14B, Structure-aware gating reaches F1 0.844, but imperfect gates do not reliably improve retention; Oracle gains remain an upper bound on 20 users.

Key insights

  • Personalized memory, when ideally applied, provides considerable improvements in Memory Retention Rate, especially under limited memory budgets, by focusing resources on genuinely critical information.
  • Understanding cross-session relational structure (Structure-aware gating) is beneficial for inferring "worth-storing" contexts but shows limited robustness to dynamic user profile shifts.
  • There is a substantial gap between the performance of an ideal (Oracle) personalized memory system and current practical gating mechanisms, indicating significant unrealized potential and challenges in accurately inferring user-specific needs.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.