Reproduce “Modeling Context in Referring Expressions” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, Tamara L. Berg
Researchers at UNC Chapel Hill developed methods to improve referring expression generation and comprehension by incorporating explicit visual comparisons between objects and enabling joint language generation. Their "visdif" model improved comprehension accuracy to 71.19% on RefCOCO Test B and their "tied" generation model significantly reduced the duplicate rate of generated expressions.
Key insights
- Explicit visual comparisons between a target object and other same-category objects are more effective for disambiguation than generic global image context.
- Focusing context modeling on visual differences to same-category distractors aligns with human cognitive processes for generating distinguishing descriptions.
- Jointly generating expressions for all objects of a given category in an image helps reduce ambiguity and improve linguistic coherence across descriptions.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.