Reproduce “BREEN: Bridge Data-Efficient Encoder-Free Multimodal Learning with Learnable Queries” on OpenResearch
Turn your favorite coding agents into research agents that review literature, develop hypotheses, and run experiments.
Universal app for Apple silicon and Intel. Requires macOS 11 or later.
About this paper
Tianle Li, Yongming Rao, Winston Hu, Yu Cheng
BREEN addresses image questions without a vision encoder at inference by using CLIP features during training to supervise query tokens between image and text tokens. An image expert processes these tokens. On ten benchmarks, it reports 60.2 normalized average score versus Mono-InternVL’s 58.1, with 9M versus 1.3B pretraining pairs; differing model sizes and SFT data make this an uncontrolled comparison.
Summary by alphaXiv.
Tip: change arxiv.org to autoarxiv.org in any arXiv abstract link to open the paper here.