Reproduce “Any Channel Count, One Token Stream: Incorporating Spatial Audio Understanding and Reasoning into Large Audio-Language Models” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yuhuan You, Wenxiang Guo, Zhifang Guo, Yuxuan Wang, Jin Xu

SpheraAudio combines sound-content tokens with time-aligned spatial tokens from multichannel audio to answer questions about sources, directions, and distance. Its spatial and audio tokens are interleaved in a language-model sequence, with positional encoding marking time and modality. On SpheraQA, its 4-billion-parameter model scored 60.18%, versus 41.65% for Spatial-Omni-7B; this whole-answer score covers varied questions, not reasoning alone.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.