Reproduce “Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jinxing Zhou, Dan Guo, Yiran Zhong, Meng Wang
A new method, VAPLAN, improves weakly-supervised Audio-Visual Video Parsing by leveraging large-scale pre-trained models (CLIP and CLAP) to generate segment-level pseudo labels, which are then exploited with a richness-aware loss and denoised for refinement. The approach achieves new state-of-the-art results on the LLP dataset, showing, for instance, a +14.4% event-level F-score for visual events over the baseline.
Key insights
- Large-scale pre-trained models like CLIP and CLAP can be effectively used in a zero-shot manner to generate high-quality, fine-grained segment-level pseudo labels for both visual and audio modalities.
- Segment-level pseudo labels provide a significantly richer supervisory signal than video-level labels, effectively mitigating cross-modal interference and enabling precise temporal localization.
- A "richness-aware" loss that considers the distribution of event categories per segment and segments per event can more effectively exploit fine-grained pseudo labels.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.