Reproduce “Geometry Attention Transformer with Position-aware LSTMs for Image Captioning” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Chi Wang, Yulin Shen, Luping Ji
The Geometry Attention Transformer (GAT) enhances image captioning by systematically integrating explicit geometry and position-aware mechanisms into the Transformer architecture. This model uses a Geometry Self-attention Refiner in its encoder and Position-LSTMs in its decoder, achieving a CIDEr score of 130.5% on MS COCO and 68.0% on Flickr30k, outperforming advanced baselines.
Key insights
- Explicitly integrating geometry features into the self-attention mechanism significantly improves the encoder's visual representation and the overall descriptive accuracy.
- Dynamic positional encodings generated by LSTMs provide more expressive and adaptive sequence order information to the decoder, enhancing its ability to model word relationships and spatial references.
- Concatenating appearance and geometry features for attention queries and keys is more effective than element-wise addition for multimodal feature integration.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.