Reproduce “Show, Edit and Tell: A Framework for Editing Image Captions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Fawaz Sammani, Luke Melas-Kyriazi
A framework for editing image captions, named 'Show, Edit and Tell,' refines existing captions rather than generating new ones, addressing issues like factual errors and repetitions, and achieving state-of-the-art performance on the MS COCO dataset.
Key insights
- Editing an existing sequence can be more effective than generating one from scratch, allowing the model to focus on refining details rather than learning overall sentence structure.
- A Selective Copy Memory Attention (SCMA) mechanism, employing non-differentiable indexing with a re-parametrization trick, enables efficient and precise copying of specific words or concepts from an input caption.
- Optimizing on the Mean Squared Error (MSE) between generated and ground truth hidden states provides a novel and effective objective for improving semantic alignment in sequence generation.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.