Reproduce “Not Every Call Needs a Frontier Model: Per-Call-Site Evaluation of Small Language Models in a Deployed Agentic Home-Automation System” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Panagiotis Kasnesis, Christos Chatzigeorgiou, Lazaros Toumanidis, Amalia Contiero Syropoulou
The authors evaluate static model assignments across five call sites in a deployed home-automation assistant, matching models to routing, actuation, planning, and code generation. They compare live configurations and test 280 benchmark cases. All-local routing scores 91.8% versus 95.4% for all-hosted; hosting planning and code generation matches all-hosted on 39/43 cases at $0.70 versus $2.47, in one installation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.