Reproduce “Transforming Visual Scene Graphs to Image Captions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Xu Yang, Jiawei Peng, Zihua Wang, Haiyang Xu, Qinghao Ye, Chenliang Li, Songfang Huang, Fei Huang and 2 more
This research introduces TFSGC, a model for transforming visual scene graphs into descriptive image captions using a homogeneous Transformer-based architecture. It designs a Graph Neural Network (GNN) and a Mixture-of-Expert (MOE) decoder based on Multi-Head Attention to selectively utilize object, attribute, and relation information, achieving competitive performance on the MS-COCO dataset with a CIDEr-D score of 139.5 using VinVL features.
Key insights
- Designing Graph Neural Networks (GNNs) using Multi-Head Attention (MHA) allows for a homogeneous encoder-decoder architecture, simplifying model training and reducing complexity.
- Learnable type embeddings and masked MHA can effectively encode graph topology and node type information within a linearized sequence for graph operations.
- A Mixture-of-Expert (MOE) decoder can selectively leverage different types of semantic information (objects, attributes, relations) from scene graphs to generate more descriptive captions by dynamically routing to specialized attention layers.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.