Reproduce “On Distinctive Image Captioning via Comparing and Reweighting” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Jiuniu Wang, Wenjia Xu, Qingzhong Wang, Antoni B. Chan
A novel training framework, Distinctiveness via Comparing and Reweighting (DCR), is introduced alongside the CIDErBtw metric to generate image captions that are more distinctive and accurate. The framework, applied to various baselines, achieves a CIDEr score of 128.53 and a CIDErBtw of 83.98 with a Transformer+SCST model, and improves human accuracy in an image retrieval task to 79.8%.
Key insights
- Quantifying distinctiveness by comparing a target image's caption to those of semantically similar images (via CIDErBtw) is an effective approach to guide model training and evaluation.
- Reweighting ground-truth captions and incorporating CIDErBtw directly into the reward function can simultaneously improve both distinctiveness and overall caption accuracy.
- Explicitly encouraging the use of low-frequency words and leveraging negative samples from similar images effectively guides models to generate more specific and less generic descriptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.