Reproduce “PersonalHomeBench: Evaluating Agents in Personalized Smart Homes” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Manasa Bharadwaj, Yolanda Liu, InJung Yang, Sungil Kim, Nikhil Verma, KoKeun Kim, Kevin Ferreira, YoungJoon Kim

LG Toronto AI Lab and LG Electronics researchers developed PersonalHomeBench, a new benchmark to assess foundation models as personalized smart home agents, integrating realistic household contexts and diverse tasks. Evaluations revealed consistent performance degradation across models when transitioning from full context to tool-mediated execution, highlighting brittle tool usage and difficulties in generating executable, preference-aligned plans.

Key insights

  • Effective tool interaction and coordination pose a significant bottleneck for current agentic systems, consistently leading to substantial performance degradation when transitioning from full context to tool-mediated execution.
  • Model scale alone does not close the performance gap between tool-less and tool-mediated execution, suggesting that primary failures originate from interaction-level reasoning and tool coordination, rather than insufficient model capacity.
  • While multimodal inputs can aid reactive task performance under full context, their benefits largely diminish in agentic settings for both reactive and proactive tasks, indicating that improved perception alone does not resolve challenges in tool use and long-horizon planning.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.