Reproduce “Beyond Recall Accuracy: Evaluating Integrity and Crash Continuity in Persistent Memory for Tool-Using Language Agents” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Javad Taghia
AtMem evaluates persistent memory for authority and recovery after interrupted tool actions, not recall alone. It filters memories by eligibility, records operation identities and receipts, and preserves uncertainty when a response is missing. In one controlled retail interruption, AtMem made zero repeat public tool calls versus one for baseline and observer-only configurations; the destination prevented duplicate effects.
Key insights
- Relevance alone cannot confer authority: provenance and authorization must be checked before retrieval ranking, and taint should persist into derived summaries or procedures.
- Operational continuity requires preserving uncertainty about an interrupted action; when neither reconciliation nor safe idempotent retry is available, stopping for confirmation is safer than issuing another call.
- Telemetry and execution authority can be separated: AtFlows observed events but did not change AtMem’s recovery decision, while the benchmark’s N/R category prevented unsupported authorization claims.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.