Reproduce “Tri-PvP: Exposing Modality Bias in Omni-Modal Large Language Models through Perceptual-Propositional Evidence Conflicts” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Yen-Ting Piao, Shu-Yun Chen, Chin-Hui Chu, Chun-Wei Chen, Shih-Yun Shan Kuan, Hung-yi Lee, Yun-Nung Chen

The authors introduce TRI-PvP, a benchmark that separates modality preference from evidence-form preference by varying whether image and audio inputs provide perceptual signals or propositions while labels remain matched. Across 8,000 samples, five models follow the image as the leading single-source answer in 18 of 20 model-condition combinations. This benchmark result is bounded by balanced labels and controlled conditions.

Key insights

  • Modality bias in OLLMs is systematically modulated by the form of evidence; models exhibit a stronger preference for perceptual visual evidence but show increased reliance on propositional audio evidence.
  • Modality-bias information is linearly decodable from early hidden layers of OLLMs, indicating that these preferences are deeply embedded within the model's internal representations, not merely surface-level output phenomena.
  • While input modality order can influence specific positional biases, the core findings regarding dominant image bias, weak audio bias, and evidence-form asymmetry remain consistent across different input permutations.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.