Reproduce “Leveraging Foundation Models for Zero-Shot IoT Sensing” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Dinghao Xue, Xiaoran Fan, Tao Chen, Guohao Lan, Qun Song

This paper presents a method for zero-shot human activity recognition on edge Internet of Things (IoT) devices, utilizing Foundation Models to classify unseen activities from mmWave, IMU, and Wi-Fi signals. The approach combines a novel prompt engineering strategy with bias mitigation techniques, achieving improved open-set detection and higher unseen class accuracy in generalized zero-shot learning across multiple IoT datasets.

Key insights

  • Foundation models, even when not natively trained on IoT signals, can be effectively leveraged for zero-shot sensing if properly aligned with sensor data.
  • Combining learnable soft prompts with domain-knowledge-infused hard prompts via a cross-attention mechanism is crucial for bridging the semantic gap between foundation models and physics-driven IoT signals.
  • Addressing seen-class bias through generative data augmentation for fine-tuning and explicit open-set detection is critical for robust generalized zero-shot learning performance in IoT.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.