Reproduce “Teaching Physical Awareness to LLMs through Sounds” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Weiguo Wang, Andy Nie, Wenrui Zhou, Yi Kai, Chengchen Hu

ACORN, a framework from NIO and Peking University, enables Large Language Models to acquire physical awareness from acoustic signals by training them on a physics-based synthesized dataset. It leverages a novel audio encoder to capture both magnitude and phase information, demonstrating enhanced performance over baselines and practical viability in real-world vehicle environments for tasks like Doppler estimation and sound source localization.

Key insights

  • Received sound signals can be effectively decomposed into a sound source and the physical channel's impulse response, enabling scalable and cost-effective synthesis of diverse, annotated training data for complex acoustic phenomena.
  • Explicitly capturing phase information, in addition to magnitude, within the audio encoding process is crucial for LLMs to accurately discern subtle physical effects like Doppler shifts, multipath reflections, and spatial relationships.
  • Teaching LLMs physical awareness through acoustic data, derived from a physics-based simulator, allows them to reason about dynamic effects like motion and spatial relationships that are not readily accessible through text or visual modalities alone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.