Reproduce “Unbiasing through Textual Descriptions: Mitigating Representation Bias in Video Benchmarks” on OpenResearch

About this paper

Nina Shvetsova, Arsha Nagrani, Bernt Schiele, Hilde Kuehne, Christian Rupprecht

The research introduces Unbiasing through Textual Descriptions (UTD), an automated pipeline that leverages Vision-Language Models and Large Language Models to identify and mitigate representation biases in video benchmarks. It quantitatively reveals pervasive object and single-frame biases across 12 popular datasets, creating debiased test sets (UTD-splits) that demonstrate state-of-the-art models' reliance on these shortcuts, with up to 73% of samples in UCF101 found to be object-biased.

Key insights

• Representation biases, particularly object and single-frame biases, are pervasive in popular video benchmarks, leading to an overestimation of model capabilities as models can achieve high performance by exploiting simple cues.
• Dataset bias (spurious correlations learned from training data) is consistently more dominant than common sense bias across analyzed benchmarks and concept categories.
• Current state-of-the-art video models heavily rely on these biases, as demonstrated by significant performance drops when evaluated on the newly created UTD-debiased test sets.

Summary by alphaXiv.