Reproduce “Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognition” on OpenResearch

About this paper

Geo Ahn, Inwoong Lee, Taeoh Kim, Minho Shim, Dongyoon Wee, Jinwoo Choi

Researchers from Kyung Hee University and NAVER Cloud introduced RCORE, a framework to mitigate object-driven shortcuts in Zero-Shot Compositional Action Recognition by encouraging temporally grounded verb learning and reducing reliance on co-occurrence biases. RCORE improved unseen composition accuracy by up to 7.0 points on EK100-com and yielded positive compositional gaps on unseen data, demonstrating more robust generalization to novel actions.

Key insights

• Object-driven shortcuts are a major failure mode in ZS-CAR, causing models to over-rely on static object features instead of temporal dynamics for verb prediction.
• The problem arises from a combination of data-driven biases (sparse/skewed compositional supervision) and model-driven biases (asymmetric learning difficulty between verbs and objects).
• Explicitly regularizing against co-occurrence priors and enforcing temporal order sensitivity in verb representations are effective strategies to mitigate shortcut learning and improve compositional generalization.

Summary by alphaXiv.