Reproduce “VehicleMemBench: An Executable Benchmark for Multi-User Long-Term Memory in In-Vehicle Agents” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuhao Chen, Yi Xu, Xinyun Ding, Xiang Fang, Shuochen Liu, Luxi Lin, Qingyu Zhang, Ya Li and 2 more

VehicleMemBench evaluates whether in-car agents can use occupants’ preferences to operate vehicle controls in a simulator. It combines long, multi-user histories with delayed queries across 500 tasks and 50 scenarios. Agents must handle conflicting, context-dependent, and changing preferences using 23 modules and 111 APIs. Gemini-3-Pro-Preview achieved 90.60% exact state match with Gold Memory versus 64.80% with Recursive Summarization, a 25.8 percentage-point drop; this measures simulator performance, not real-world driving.

Key insights

  • Autonomous memory construction and retrieval represent a significant bottleneck for current LLM agents, consistently causing a substantial performance drop compared to scenarios where correct memory is provided.
  • General-purpose memory systems often underperform simpler, domain-specific baselines, suggesting they are not well-suited for the complex, dynamic requirements of in-vehicle multi-user long-term memory.
  • Handling 'Conditional Constraints' (context-sensitive preferences) is the most challenging memory type for agents, indicating difficulties in modeling context-dependent preferences that vary with external conditions.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.