Reproduce “Attentive AV-FusionNet: Audio-Visual Quality Prediction with Hybrid Attention” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Ina Salaj, Arijit Biswas
Researchers at Dolby Germany GmbH developed Attentive AV-FusionNet, a deep learning model that accurately predicts audio-visual quality by integrating state-of-the-art unimodal features with hybrid attention mechanisms. The model achieved a Pearson correlation of 0.97 and RMSE of 0.22 on an internal dataset, while also providing content-dependent insights into modality importance.
Key insights
- Leveraging rich internal features from state-of-the-art unimodal predictors (like VMAF and GML) is crucial for accurate audio-visual quality prediction.
- Hybrid attention mechanisms, specifically bidirectional cross-attention and self-attention, effectively capture complex, content-dependent interactions between audio and video modalities.
- Content-aware modality relevance can be estimated, providing actionable insights beyond a single quality score for adaptive streaming optimizations.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.