Reproduce “Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, Jianlong Fu
Pixel-BERT replaces detector-region inputs with spatial feature cells from a CNN, which are combined with text tokens in a shared Transformer for image–language learning. The cells are pooled CNN features, not raw pixels. On VQA 2.0 test-dev, its ResNeXt-152 model reports 74.45 VQA-score, versus UNITER’s 72.27; this system comparison does not isolate the representation’s effect.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.