Reproduce “Composing Object Relations and Attributes for Image-Text Matching” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Khoi Pham, Chuong Huynh, Ser-Nam Lim, Abhinav Shrivastava
CORA (Composition model for Object Relations and Attributes) introduces a dual-encoder model for image-text matching that enhances text encoding by representing captions as scene graphs and processing them with Graph Attention Networks. This approach improves image-text retrieval performance on Flickr30K by 5.4 RSUM points and on MS-COCO 5K by 13.7 RSUM points over previous state-of-the-art models, while maintaining the computational efficiency of dual-encoder architectures.
Key insights
- Representing captions as scene graphs enables a dual-encoder model to capture rich semantic structures, including object-attribute and object-object relationships, which traditional sequence models often miss.
- Graph Attention Networks (GATs) are effective for encoding hierarchical information within scene graphs, allowing for richer object and scene embeddings from textual data.
- Integrating structured text encoding via scene graphs into a dual-encoder architecture can bridge the gap between the efficiency of dual-encoders and the fine-grained alignment capabilities of more computationally intensive cross-attention models.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.