Reproduce “OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understanding” on OpenResearch

Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.

Universal app for Apple silicon and Intel. Requires macOS 11 or later.

About this paper

Jieer Ouyang, Xiaoneng Xiang, Zheng Wang, Yangkai Ding

Researchers at Huawei Singapore Research Center developed OTTER, an open-set multi-label tagging framework that effectively harmonizes fixed predefined categories with adaptable open tags. This system demonstrated superior multi-label prediction accuracy on proprietary datasets, achieving F1 scores of 0.81 and 0.75, and notably excelled in open-set tagging with F1 scores up to 0.99.

Key insights

  • A dual-layer tagging paradigm can effectively balance the consistency of fixed taxonomies with the adaptability required for open-vocabulary labels in dynamic content environments.
  • Robust multi-modal feature fusion via element-wise addition and the integration of rich semantic cues during training are crucial for enhancing discriminative capacity in open-set tagging scenarios.
  • Achieving high performance on dynamically defined open-set labels, while maintaining competitive accuracy on predefined ones, demonstrates the model's ability to handle novel categories and semantic variability without compromise.

Summary by alphaXiv.

Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.