Reproduce “Generation and Comprehension of Unambiguous Object Descriptions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, Kevin Murphy
This research from Google Inc., UCLA, University of Oxford, and Johns Hopkins University introduced a discriminative training approach for deep learning models that generate and comprehend unambiguous object descriptions. The method, leveraging a new large-scale dataset called Google Refexp (G-Ref), achieved a 4.5% absolute improvement in human preference for generated descriptions compared to a maximum likelihood baseline, and boosted comprehension Precision@1 on the UNC-Ref-Test set to 0.700.
Key insights
- Explicitly modeling a 'listener's' comprehension ability during training (via discriminative MMI objectives) significantly improves the generated descriptions' unambiguousness and utility.
- Using 'hard negative' examples (confusing objects of the same class) during discriminative training is crucial for teaching the model to produce highly discriminative referring expressions.
- Focusing on the referring expression task offers an objective and measurable framework for evaluating vision-language systems, overcoming the subjectivity challenges of general image captioning.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.