Reproduce “SensorQA: A Question Answering Benchmark for Daily-Life Monitoring” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Benjamin Reichman, Xiaofan Yu, Lanxiang Hu, Jack Truxal, Atishay Jain, Rushil Chandrupatla, Tajana Šimunić Rosing, Larry Heck
SensorQA introduces the first human-created question-answering benchmark designed for interpreting long-term, multi-modal sensor data from daily life monitoring. Evaluations on this benchmark reveal that state-of-the-art AI models achieve limited accuracy, peaking at 28% exact-match, and exhibit impractically high latency exceeding 57 seconds on edge devices for such complex sensor-text reasoning tasks.
Key insights
- A substantial gap exists between current AI model capabilities and the desired performance and efficiency for effective human-sensor QA interactions, especially in interpreting long-term, diverse sensor data.
- Current models struggle with effective sensor-text fusion, with Sensor+Text baselines often performing worse than or comparably to Text-only baselines, suggesting raw sensor data integration can confound models.
- Fine-tuning open-source models consistently outperforms few-shot learning with closed-source models, highlighting the critical need for task-specific adaptation on the SensorQA dataset.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.