Reproduce “PanLUNA: An Efficient and Robust Query-Unified Multimodal Model for Edge Biosignal Intelligence” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Marija Zelic, Anna Tegon, Yawei Li, Thorir Mar Ingolfsson, Luca Benini
PanLUNA, developed by ETH Zürich, introduces a compact query-unified multimodal foundation model capable of jointly processing EEG, ECG, and PPG biosignals within a single shared encoder. The model achieves competitive performance on various tasks, demonstrates robustness to missing modalities, and is deployable on ultra-low-power microcontrollers, performing a 10-second 12-lead ECG inference in 325.6 ms.
Key insights
- A single shared encoder with query-level attention can effectively integrate multiple biosignal modalities (EEG, ECG, PPG) by learning functionally specialized queries based on sensor-type embeddings.
- Multimodal pretraining, even without large paired datasets, enables the creation of highly compact models (5.4M parameters) that achieve performance comparable to or better than significantly larger existing models.
- Aggressive quantization-aware training (down to INT2 weights) can preserve high model accuracy, facilitating deployment of complex multimodal physiological FMs on ultra-low-power edge devices with real-time inference capabilities.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.